AlphaTcl

Hi Bernard,

this is fantastic news!  Long live Alpha!

I agree that not being open source has been a big
obstruction for the adoption Alpha deserves, but
hasn't had since the days it was distributed with OzTeX.
Lack of unicode support is another showstopper for many
people (ironically mostly people who never write in any
other language than English, but who have read on the
blogs that unicode is hip).

Concerning redesign, there are several things in
AlphaTcl that I think could deserve to be reorganised,
for example prefs (in order to write to disk immediately
and only ever write values different from the default
-- such a system is used by consoleAttributes.tcl)
and package handling (it should be possible to avoid that
crazy package rebuilding system and rely instead on
standard Tcl indices, in some way).  These two issues seem
to be independent of the core, though, so it would only be
to take advantage of the moment to include some other
reforms.

Concerning the core, I think it would be great to have it
smaller, and implement more lowlevel procs in Tcl.  This is
perhaps mostly in the section on positions and text
manipulation, where many core procs could easily be
implemented in Tcl on the basis of very few core procs.

I wonder if it would be possible (and desirable) to have
some sort of link such that at any given time, the content of
a window is at the same time the value of a Tcl variable, so
that [search] could be [regexp], [getText] could be
[string range], and so on, and plain Tcl could play a more
prominent role.  Of course we would have convenience procs for
all the standard editing, but it would be closer to pure Tcl.
I suppose that in any case the content of a window must exist
as the value of a variable somewhere, just not available to
Tcl.  I don't know if this is really a good idea, for
performance  reasons, practical reasons or whatever.
Please enlighten me.

I have no experience in Cocoa programming.  But if Alpha
goes open source, perhaps that is a good opportunity to
learn a little bit!

Concerning the name, AlphaCocoa is a natural name for the
new project of course, but I hope that the name of the
application will be just Alpha.   Other great applications
such as tex and emacs don't change name each time they
are reimplemented or ported to a new platform...

Thanks again, Bernard, for taking Alpha forward!

Cheers,
Joachim.

Note: I'm perfectly happy to open-source any and all Alphatk code if
it helps any such effort. (Not sure if TkCocoa + modifications is a
good basis of this or not).

Vince.

On 26 June 2011 09:42, Bernard Desgraupes <bdesgraupes@...> wrote:
> Hi all,
>
> I'd like to make a proposal concerning the future development of the
> Alpha core.
>
> Currently AlphaX is an application which entirely relies, after two
> years of rewriting, on the Carbon framework and on the Carbon Events
> model.  Alas the Carbon technology is now considered obsolete by Apple
> which just maintains it for compatibility or legacy reasons and in
> order not to rule out many applications which have adopted this same
> model. But it has been stated clearly that no future development will
> be made in this direction and in particular there will never be a
> 64bits version of Carbon. The question even arises to know when Carbon
> will be completely dropped (it is confirmed that it is still present
> in the forthcoming Lion system, but what after that): so there is a
> risk that in some time Alpha won't run on future versions of the system.
>
> On the contrary Apple insists on having developers switch to Objective-
> C and to the Cocoa framework and puts all its energy in this new path.
> So I think it is time to bite the bullet and to go Cocoa. This implies
> a rewriting from scratch of the source code. This is obviously the
> condition sine qua none to keep in synch with new future technologies
> and, for example, to be able to develop a version of Alpha for the iPad.
>
> Here is now my proposal: as I have already suggested in the past,
> Alpha would really benefit from being an Open Source project. If it
> were an Open Source modern project, I'm sure a lot of people would be
> attracted and ready to contribute because it is a really powerful and
> fascinating tool. So I tpropose we start a new project to write from
> scrap a new core for the already existing and open sourc'ed AlphaTcl
> library. This is an approach which is comparable to the AlphaTk
> project in its time: this would be an independant project written in
> another language (Objective-C) and relying exclusively on the Cocoa
> framework and the Quartz technology for drawing.
>
> I'm ready to undertake such a challenge and, as a matter of fact, I've
> already led some experiments and made some headway. I have a minimal
> Cocoa application embedding the Tcl framework and starting AlphaTcl.
> It also embeds the ICU framework to have a complete support for
> Unicode and I have implemented a new [icu] core command to deal with
> all the Unicode-related manipulations.
>
> This new project, which could be named AlphaCocoa, might be installed
> on SourceForge to make it accessible to everybody. This would be a
> great opportunity of redesigning the interface, discussing new ideas,
> making suggestions, and probably also rethinking the structure of
> AlphaTcl itself.
>
> I'd be glad to hear opinions about these ideas.
>
> Cheers,
> Bernard
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> AlphaTcl-developers mailing list
> AlphaTcl-developers@...
> http://lists.sourceforge.net/lists/listinfo/alphatcl-developers
>

Hi Vince,

i'm glad to hear from you.
What is the current state of AlphaTk ? You mention TkCocoa, so does  
this mean that you now build AlphaTk over Tk 8.6 which is (IIUC) a  
Cocoa product.
Would you take part in the AlphaCocoa Project?

cheers,
Bernard


Le 27 juin 11 à 10:38, Vince Darley a écrit :

> Note: I'm perfectly happy to open-source any and all Alphatk code if
> it helps any such effort. (Not sure if TkCocoa + modifications is a
> good basis of this or not).
>
> Vince.

As you'll have noticed I haven't spent any significant time on Alphatk
lately, although I believe it worked ok on TkCocoa last time I tried.
I honestly don't know if I would have time to do anything significant
in the project, but you never know... The interest is there, but the
time is mostly not (my coding is mostly in Java with Eclipse these
days, although whenever I use Tcl I fire up AlphaX/Alphatk right away)

Vince.

On 27 June 2011 10:33, Bernard Desgraupes <bdesgraupes@...> wrote:
> Hi Vince,
>
> i'm glad to hear from you.
> What is the current state of AlphaTk ? You mention TkCocoa, so does this
> mean that you now build AlphaTk over Tk 8.6 which is (IIUC) a Cocoa product.
> Would you take part in the AlphaCocoa Project?
>
> cheers,
> Bernard
>
>
> Le 27 juin 11 à 10:38, Vince Darley a écrit :
>
>> Note: I'm perfectly happy to open-source any and all Alphatk code if
>> it helps any such effort. (Not sure if TkCocoa + modifications is a
>> good basis of this or not).
>>
>> Vince.
>
>

Joachim Kock skrev 2011-06-27 00.17:
> Hi Bernard,
>
> this is fantastic news!  Long live Alpha!

Ditto.

> I agree that not being open source has been a big
> obstruction for the adoption Alpha deserves, but
> hasn't had since the days it was distributed with OzTeX.
> Lack of unicode support is another showstopper for many
> people (ironically mostly people who never write in any
> other language than English, but who have read on the
> blogs that unicode is hip).

There is also the matter that an increasing number of config files one would 
like to use Alpha for poking around in are in UTF-8 (or -16). Even if the 
contents could comfortably fit in macRoman, it's not all that fun to have 
Alpha interpret the file contents that way.

> Concerning redesign, there are several things in
> AlphaTcl that I think could deserve to be reorganised,
> for example prefs (in order to write to disk immediately
> and only ever write values different from the default
> -- such a system is used by consoleAttributes.tcl)
> and package handling (it should be possible to avoid that
> crazy package rebuilding system and rely instead on
> standard Tcl indices, in some way).

Are you thinking about Alpha's package index or [auto_load] index? The 
former is not that different from the Tcl package index, and the latter is a 
standard Tcl mechanism (even if deprecated). There is of course the Tcl 
Modules mechanism for packages which does not rely on indexing, but it also 
couldn't provide all the extra information that Alpha 
Menus/Features/whatever come with.

> These two issues seem
> to be independent of the core, though, so it would only be
> to take advantage of the moment to include some other
> reforms.
>
> Concerning the core, I think it would be great to have it
> smaller, and implement more lowlevel procs in Tcl.  This is
> perhaps mostly in the section on positions and text
> manipulation, where many core procs could easily be
> implemented in Tcl on the basis of very few core procs.
>
> I wonder if it would be possible (and desirable) to have
> some sort of link such that at any given time, the content of
> a window is at the same time the value of a Tcl variable, so
> that [search] could be [regexp], [getText] could be
> [string range], and so on, and plain Tcl could play a more
> prominent role.

No, this is *not* possible (at any reasonable cost). It would force Alpha to 
store the contents of a window as a long string (it could additionally store 
the contents in other formats, but the string would be unavoidable over 
time), which makes insertions grossly inefficient.

> Of course we would have convenience procs for
> all the standard editing, but it would be closer to pure Tcl.
> I suppose that in any case the content of a window must exist
> as the value of a variable somewhere, just not available to
> Tcl.  I don't know if this is really a good idea, for
> performance  reasons, practical reasons or whatever.
> Please enlighten me.
>
> I have no experience in Cocoa programming.  But if Alpha
> goes open source, perhaps that is a good opportunity to
> learn a little bit!

Ditto. I've looked at the TkCocoa event handling code a bit (trying to 
figure out how to fix a bug, but so far without success; I haven't spent all 
that much time on it though).

One question here is how heavily objectified the code is likely to be. With 
C++ (and I strongly suspect the case would be the same with Objective-C), 
there are some programmers who go all-in for the new features, and others 
who mostly continue writing C, only using the new language's extensions when 
they need to (e.g. because some library is written in it). I believe Tcl 
projects tend to lean towards the latter. Do you, Bernard, have any ideas 
about how AlphaCocoa would lean?

Somewhat related to this, I suppose, is the matter of whether to make heavy 
use of Apple libraries (e.g., declare that the contents of a window are now 
an NSMutableAttributedString or some such), or to rather roll one's own and 
use Tcl offerings (e.g. the internals of the [text] widget) whenever 
practical. This is a tricky matter, with far-reaching consequences, and I'm 
not sure I could guess which choice would be best...

Lars Hellström

Hi Lars,


Le 27 juin 11 à 14:34, Lars a écrit :

> Joachim Kock skrev 2011-06-27 00.17:
>>
>
>> I agree that not being open source has been a big
>> obstruction for the adoption Alpha deserves, but
>> hasn't had since the days it was distributed with OzTeX.
>> Lack of unicode support is another showstopper for many
>> people (ironically mostly people who never write in any
>> other language than English, but who have read on the
>> blogs that unicode is hip).
>
> There is also the matter that an increasing number of config files  
> one would
> like to use Alpha for poking around in are in UTF-8 (or -16). Even  
> if the
> contents could comfortably fit in macRoman, it's not all that fun to  
> have
> Alpha interpret the file contents that way.

The idea is that AlphaCocoa will support all kinds of input or output  
encodings (as AlpahTk does already). It should be able to edit text  
files in almost any encoding (known to Tcl).
I think the internal buffer will be in UTF16.
I have also compiled the ICU library as a framework and embedded it  
inside the application in order to have full Unicode support (Tcl is  
incomplete in this regard).

>> I wonder if it would be possible (and desirable) to have
>> some sort of link such that at any given time, the content of
>> a window is at the same time the value of a Tcl variable, so
>> that [search] could be [regexp], [getText] could be
>> [string range], and so on, and plain Tcl could play a more
>> prominent role.
>
> No, this is *not* possible (at any reasonable cost). It would force  
> Alpha to
> store the contents of a window as a long string (it could  
> additionally store
> the contents in other formats, but the string would be unavoidable  
> over
> time), which makes insertions grossly inefficient.

I agree.


> One question here is how heavily objectified the code is likely to  
> be. With
> C++ (and I strongly suspect the case would be the same with  
> Objective-C),
> there are some programmers who go all-in for the new features, and  
> others
> who mostly continue writing C, only using the new language's  
> extensions when
> they need to (e.g. because some library is written in it). I believe  
> Tcl
> projects tend to lean towards the latter. Do you, Bernard, have any  
> ideas
> about how AlphaCocoa would lean?

My intention is to go entirely Cocoa: no half baked solutions mixing  
old C code and occasional use of Objc.
I'm progressively discovering the Cocoa framework and it is much  
easier to use than the CoreFoundation framework which I used  
extensively in the carbon version.

The only dilemma I have regularly (and already had with AlphaCarbon)  
is when it comes to decide what kind of object to use when strings,  
lists or dicts are involved: the Tcl API or the NS classes (previously  
the CF API). I tend to choose the one which causes as few conversions  
(Tcl_Obj to NS class and vice versa) as possible.


> Somewhat related to this, I suppose, is the matter of whether to  
> make heavy
> use of Apple libraries (e.g., declare that the contents of a window  
> are now
> an NSMutableAttributedString or some such), or to rather roll one's  
> own and
> use Tcl offerings (e.g. the internals of the [text] widget) whenever
> practical. This is a tricky matter, with far-reaching consequences,  
> and I'm
> not sure I could guess which choice would be best...

Concerning the text engine, I haven't decided yet at which level we  
should work. I have studied the CoreText API which seems to be  
sufficiently low-level for our needs (indeed making use of  
NSMutableAttributedStrings etc.) but I have no idea of how efficient  
it is (in terms of performance). Maybe ATSUI is a better choice but it  
is not clear to me what is the future of the ATSUI framework.

Lots of open questions...
If we start this project on SourceForge, I will install a trac  
environment with a wiki in order to discuss all the aspects of the  
project, the design choices, and so on.


cheers,
Bernard

Hi all

On 26/06/2011, at 10:42 , Bernard Desgraupes wrote:

Hi all,

I'd like to make a proposal concerning the future development of the
Alpha core.

This is good. However, I propose to distill two separate issues out of this:

1) Making AlphaX open source
2) Using Cocoa or Objective C as the new basis

I am fully in favour of making AlphaX open source and I trust, that we can easily agree on this.

The 2nd issue is IMHO a bit more complicated, as I try to explain below:


Currently AlphaX is an application which entirely relies, after two
years of rewriting, on the Carbon framework and on the Carbon Events
model.  Alas the Carbon technology is now considered obsolete by Apple
which just maintains it for compatibility or legacy reasons and in
order not to rule out many applications which have adopted this same
model. But it has been stated clearly that no future development will
be made in this direction and in particular there will never be a
64bits version of Carbon. The question even arises to know when Carbon
will be completely dropped (it is confirmed that it is still present
in the forthcoming Lion system, but what after that): so there is a
risk that in some time Alpha won't run on future versions of the system.

On the contrary Apple insists on having developers switch to Objective-
C and to the Cocoa framework and puts all its energy in this new path.
So I think it is time to bite the bullet and to go Cocoa. This implies
a rewriting from scratch of the source code. This is obviously the
condition sine qua none to keep in synch with new future technologies
and, for example, to be able to develop a version of Alpha for the iPad.


I am happy with biting bullets as long as we don't joke on it. I mean we need a good chance and realistic perspectives to be able to really accomplish this. Who all are helping Bernard on that? Of course, Bernard, I greatly appreciate your immense capabilities, enthusiasm etc. I really do. But I would feel better, if not too much burden will

Then I'd like to draw your attention to the fact that Objective C is not good enough to move to the iPad. The basis is quite different (Mac OS X vs. iOS). While it is true that underneath iOS is nothing else than a Unix, to access it directly and hereby hacking/circumventing iOS is not the way to go and invites only troubles.

Moreover, I have an iPad but I doubt I would use it ever for any heavy editing with AlphaX. This is exactly where I have to resume work on my Mac.  For simple viewing of source code or changing a tiny little bit, we have already nice text based Apps such as SimpleNote. If your desire should be to use AlphaX as a tcl engine to do tasks in the background, that's an entire different thing. But since iOS gives you no full file system, I doubt there are many things one could do with such AlphaX services.


Here is now my proposal: as I have already suggested in the past,
Alpha would really benefit from being an Open Source project. If it
were an Open Source modern project, I'm sure a lot of people would be
attracted and ready to contribute because it is a really powerful and
fascinating tool. So I tpropose we start a new project to write from
scrap a new core for the already existing and open sourc'ed AlphaTcl
library. This is an approach which is comparable to the AlphaTk
project in its time: this would be an independant project written in
another language (Objective-C) and relying exclusively on the Cocoa
framework and the Quartz technology for drawing.

I'm ready to undertake such a challenge and, as a matter of fact, I've
already led some experiments and made some headway. I have a minimal
Cocoa application embedding the Tcl framework and starting AlphaTcl.
It also embeds the ICU framework to have a complete support for
Unicode and I have implemented a new [icu] core command to deal with
all the Unicode-related manipulations.

I agree but incidentally, I regularly enjoy AlphaX not being unicode. I regularly encounter utf encoded texts that won't be imported by BibDesk. AlphaX is the quickest remedy for that, precisely because it does **not** work with unicode. If AlphaX is going unicode, I hope there will be a clear philosophy and converting means for that, similar to what we currently have with respect to Mac, Unix, and IBM PC text files.


This new project, which could be named AlphaCocoa, might be installed
on SourceForge to make it accessible to everybody. This would be a
great opportunity of redesigning the interface, discussing new ideas,
making suggestions, and probably also rethinking the structure of
AlphaTcl itself.

I also don't like the name AlphaCocoa. AlphaX was fine with me because Mac OS X is more likely to stay a bit with us (I hope at least ;-) ). If a generic new Alpha should emerge (unification with AlphaTk), the old name 'Alpha' might be quite appropriate, leaving a phase of branching behind us.

With respect to the implementation philosophy of a Cocoa version of AlphaX. I suggest the following balancing:

1) Minimum use of Cocoa, basically confined to the GUI aspects (use of Interface Builder) and event control
2) Rest of code in C as much as possible, reusing current AlphaX code, basically only abandoning of Carbon event handling
3) Making good use of AlphaTk by relying as much as possible on Tcl/Tk seems to me to be a quite a good and possible fruitful idea, inasmuch this helps to minimise the recoding effort of the binary kernel (as described under point 2) above). Vince, nice to hear from you: Is really everything of AlphaTk in Tcl/Tk, or is there not some other "binary" kernel? Sorry, have never looked at the source code. If there is some binary kernel, what is the ratio?


I'd be glad to hear opinions about these ideas.

Cheers,
Bernard

Cheers,
Andreas

ETH Zurich
Prof. Dr. Andreas Fischlin
Systems Ecology - Institute of Integrative Biology
CHN E 21.1
Universitaetstrasse 16
8092 Zurich
SWITZERLAND

andreas.fischlin@...>
 http://www.sysecol.ethz.ch<http://www.sysecol.ethz.ch>

+41 44 633-6090 phone
+41 44 633-1136 fax
+41 79 221-4657 mobile

             Make it as simple as possible, but distrust it!
________________________________________________________________________

Hi all,

Sorry, previous E-mail was sent prematurely and contains an incomplete sentence. Ignore it and read only this one.

On 26/06/2011, at 10:42 , Bernard Desgraupes wrote:

Hi all,

I'd like to make a proposal concerning the future development of the
Alpha core.

This is good. However, I propose to distill two separate issues out of this:

1) Making AlphaX open source
2) Using Cocoa or Objective C as the new basis

I am fully in favour of making AlphaX open source and I trust, that we can easily agree on this.

The 2nd issue is IMHO a bit more complicated, as I try to explain below:


Currently AlphaX is an application which entirely relies, after two
years of rewriting, on the Carbon framework and on the Carbon Events
model.  Alas the Carbon technology is now considered obsolete by Apple
which just maintains it for compatibility or legacy reasons and in
order not to rule out many applications which have adopted this same
model. But it has been stated clearly that no future development will
be made in this direction and in particular there will never be a
64bits version of Carbon. The question even arises to know when Carbon
will be completely dropped (it is confirmed that it is still present
in the forthcoming Lion system, but what after that): so there is a
risk that in some time Alpha won't run on future versions of the system.

On the contrary Apple insists on having developers switch to Objective-
C and to the Cocoa framework and puts all its energy in this new path.
So I think it is time to bite the bullet and to go Cocoa. This implies
a rewriting from scratch of the source code. This is obviously the
condition sine qua none to keep in synch with new future technologies
and, for example, to be able to develop a version of Alpha for the iPad.


I am happy with biting bullets as long as we don't joke on it. I mean we need a good chance and realistic perspectives to be able to really accomplish this. Who all are helping Bernard on that? Of course, Bernard, I greatly appreciate your immense capabilities, enthusiasm etc. I really do. But I would feel better, if not too much burden will rest on your Atlas shoulders.

Then I'd like to draw your attention to the fact that Objective C is not good enough to move to the iPad. The basis is quite different (Mac OS X vs. iOS). While it is true that underneath iOS is nothing else than a Unix, to access it directly and hereby hacking/circumventing iOS is not the way to go and invites only troubles.

Moreover, I have an iPad but I doubt I would use it ever for any heavy editing with AlphaX. This is exactly where I have to resume work on my Mac.  For simple viewing of source code or changing a tiny little bit, we have already nice text based Apps such as SimpleNote. If your desire should be to use AlphaX as a tcl engine to do tasks in the background, that's an entire different thing. But since iOS gives you no full file system, I doubt there are many things one could do with such AlphaX services.


Here is now my proposal: as I have already suggested in the past,
Alpha would really benefit from being an Open Source project. If it
were an Open Source modern project, I'm sure a lot of people would be
attracted and ready to contribute because it is a really powerful and
fascinating tool. So I tpropose we start a new project to write from
scrap a new core for the already existing and open sourc'ed AlphaTcl
library. This is an approach which is comparable to the AlphaTk
project in its time: this would be an independant project written in
another language (Objective-C) and relying exclusively on the Cocoa
framework and the Quartz technology for drawing.

I'm ready to undertake such a challenge and, as a matter of fact, I've
already led some experiments and made some headway. I have a minimal
Cocoa application embedding the Tcl framework and starting AlphaTcl.
It also embeds the ICU framework to have a complete support for
Unicode and I have implemented a new [icu] core command to deal with
all the Unicode-related manipulations.

I agree but incidentally, I regularly enjoy AlphaX not being unicode. I regularly encounter utf encoded texts that won't be imported by BibDesk. AlphaX is the quickest remedy for that, precisely because it does **not** work with unicode. If AlphaX is going unicode, I hope there will be a clear philosophy and converting means for that, similar to what we currently have with respect to Mac, Unix, and IBM PC text files.


This new project, which could be named AlphaCocoa, might be installed
on SourceForge to make it accessible to everybody. This would be a
great opportunity of redesigning the interface, discussing new ideas,
making suggestions, and probably also rethinking the structure of
AlphaTcl itself.

I also don't like the name AlphaCocoa. AlphaX was fine with me because Mac OS X is more likely to stay a bit with us (I hope at least ;-) ). If a generic new Alpha should emerge (unification with AlphaTk), the old name 'Alpha' might be quite appropriate, leaving a phase of branching behind us.

With respect to the implementation philosophy of a Cocoa version of AlphaX. I suggest the following balancing:

1) Minimum use of Cocoa, basically confined to the GUI aspects (use of Interface Builder) and event control
2) Rest of code in C as much as possible, reusing current AlphaX code, basically only abandoning of Carbon event handling
3) Making good use of AlphaTk by relying as much as possible on Tcl/Tk seems to me to be a quite a good and possible fruitful idea, inasmuch this helps to minimise the recoding effort of the binary kernel (as described under point 2) above). Vince, nice to hear from you: Is really everything of AlphaTk in Tcl/Tk, or is there not some other "binary" kernel? Sorry, have never looked at the source code. If there is some binary kernel, what is the ratio?


I'd be glad to hear opinions about these ideas.

Cheers,
Bernard

Cheers,
Andreas

ETH Zurich
Prof. Dr. Andreas Fischlin
Systems Ecology - Institute of Integrative Biology
CHN E 21.1
Universitaetstrasse 16
8092 Zurich
SWITZERLAND

andreas.fischlin@...>
 http://www.sysecol.ethz.ch<http://www.sysecol.ethz.ch>

+41 44 633-6090 phone
+41 44 633-1136 fax
+41 79 221-4657 mobile

             Make it as simple as possible, but distrust it!
________________________________________________________________________

Hi Andreas,

thanks for your remarks. A few comments below..


Le 27 juin 11 à 15:51, Andreas a écrit :
>
>
> This is good. However, I propose to distill two separate issues out  
> of this:
>
> 1) Making AlphaX open source
> 2) Using Cocoa or Objective C as the new basis
>
> I am fully in favour of making AlphaX open source and I trust, that  
> we can easily agree on this.

Whatever the name of the project will finally be, AlphaX as it stands  
is not open source and I don't know if it can be made so.


>
> Then I'd like to draw your attention to the fact that Objective C is  
> not good enough to move to the iPad. The basis is quite different  
> (Mac OS X vs. iOS). While it is true that underneath iOS is nothing  
> else than a Unix, to access it directly and hereby hacking/ 
> circumventing iOS is not the way to go and invites only troubles.

That's a good point. I have never looked into iOS.

In fact I was rather concerned by not seeing Alpha left on the side of  
the road. It is already a miracle that it still works with a text  
engine based on QuickDraw. ICurrently Alpha would never be accepted on  
the AppStore.

> I agree but incidentally, I regularly enjoy AlphaX not being  
> unicode. I regularly encounter utf encoded texts that won't be  
> imported by BibDesk. AlphaX is the quickest remedy for that,  
> precisely because it does **not** work with unicode. If AlphaX is  
> going unicode, I hope there will be a clear philosophy and  
> converting means for that, similar to what we currently have with  
> respect to Mac, Unix, and IBM PC text files.

The regular complaint I hear about Alpha is that it is not able to  
edit UTF8 files, although, as Joachim pointed out, it very often come  
from people who don't use any symbol outside of the ASCII encoding!



>
> This new project, which could be named AlphaCocoa, might be installed
> on SourceForge to make it accessible to everybody. This would be a
> great opportunity of redesigning the interface, discussing new ideas,
> making suggestions, and probably also rethinking the structure of
> AlphaTcl itself.
>
> I also don't like the name AlphaCocoa. AlphaX was fine with me  
> because Mac OS X is more likely to stay a bit with us (I hope at  
> least ;-) ). If a generic new Alpha should emerge (unification with  
> AlphaTk), the old name 'Alpha' might be quite appropriate, leaving a  
> phase of branching behind us.

I proposed AlphaCocoa as a name for the whole project, not the name of  
the application. I agree it would be great to be allowed to call it  
Alpha. We'd need the Pete Keleher's agreement.
Otherwise it could be AlphaC or AlphaXC (?!).


> With respect to the implementation philosophy of a Cocoa version of  
> AlphaX. I suggest the following balancing:
>
> 1) Minimum use of Cocoa, basically confined to the GUI aspects (use  
> of Interface Builder) and event control
> 2) Rest of code in C as much as possible, reusing current AlphaX  
> code, basically only abandoning of Carbon event handling

This sounds good although it is not always obvious: some constants and  
symbols are defined in the carbon framework and one of the main goals  
of this migration is to abandon Carbon.
OTOH, I think there is no problem continuing to use the CoreFoundation  
framework (The CF API).

> 3) Making good use of AlphaTk by relying as much as possible on Tcl/ 
> Tk seems to me to be a quite a good and possible fruitful idea,  
> inasmuch this helps to minimise the recoding effort of the binary  
> kernel (as described under point 2) above). Vince, nice to hear from  
> you: Is really everything of AlphaTk in Tcl/Tk, or is there not some  
> other "binary" kernel? Sorry, have never looked at the source code.  
> If there is some binary kernel, what is the ratio?

There is indeed a C kernel in AlphaTk, which Vince suggested he might  
open source. I don't know exactly which framework it is based on but I  
guess it is Carbon.

cheers,
Bernard

Bernard Desgraupes skrev 2011-06-27 15.21:
> The idea is that AlphaCocoa will support all kinds of input or output
> encodings (as AlpahTk does already). It should be able to edit text files in
> almost any encoding (known to Tcl).
> I think the internal buffer will be in UTF16.
> I have also compiled the ICU library as a framework and embedded it inside
> the application in order to have full Unicode support (Tcl is incomplete in
> this regard).

It woefully is.

For data interchange (Tcl <-> full Unicode), you might find my half-baked 
UTF package useful. Code is at http://wiki.tcl.tk/_repo/UTF/, and I've 
written something about it on http://wiki.tcl.tk/14026. But on the other 
hand, ICU might provide the equivalent functionality as-is.

>> Somewhat related to this, I suppose, is the matter of whether to make heavy
>> use of Apple libraries (e.g., declare that the contents of a window are now
>> an NSMutableAttributedString or some such), or to rather roll one's own and
>> use Tcl offerings (e.g. the internals of the [text] widget) whenever
>> practical. This is a tricky matter, with far-reaching consequences, and I'm
>> not sure I could guess which choice would be best...
>
> Concerning the text engine, I haven't decided yet at which level we should work. I have studied the CoreText API which seems to be sufficiently low-level for our needs (indeed making use of NSMutableAttributedStrings etc.) but I have no idea of how efficient it is (in terms of performance). Maybe ATSUI is a better choice but it is not clear to me what is the future of the ATSUI framework.

The laziest approach is probably to use the Cocoa equivalent of the [text] 
widget. One major concern with that is whether it scales to multi-megabyte 
files (I recall the pre-Carbon counterpart had 32K as hard limit). Another 
concern is that it might bring us lots of features we don't want, resulting 
in featuritis; attributed strings simply scream "I live in a word processor".

Lars Hellström

Fischlin Andreas skrev 2011-06-27 15.51:
> Then I'd like to draw your attention to the fact that Objective C is not good enough to move to the iPad. The basis is quite different (Mac OS X vs. iOS). While it is true that underneath iOS is nothing else than a Unix, to access it directly and hereby hacking/circumventing iOS is not the way to go and invites only troubles.
>
> Moreover, I have an iPad but I doubt I would use it ever for any heavy editing with AlphaX. This is exactly where I have to resume work on my Mac.  For simple viewing of source code or changing a tiny little bit, we have already nice text based Apps such as SimpleNote. If your desire should be to use AlphaX as a tcl engine to do tasks in the background, that's an entire different thing. But since iOS gives you no full file system, I doubt there are many things one could do with such AlphaX services.

Remember that Tcl (thanks in no small part to Vince) has virtual file 
systems! If basic operations such as [save] were written on top of the Tcl 
APIs, then it wouldn't matter how feeble the OS's data storage is, as long 
as it can store one blob of data.

I'd be much more concerned about Tcl as such being the show-stopper for 
iPad, since as I understand it Apple don't like apps being general enough to 
offer a programming environment.

Lars Hellström

Before I was whisked away into the great oblivion that is my current job about a decade ago, I did a little coding in the AlphaX core, here are my thoughts on the move:

Moving to Objective-C/Cocoa for the UI is probably the only way forward, so this is generally a good move. See below for my thoughts on the moving the core to Objective-C.

I believe Pete's concern about open sourcing the code was that he didn't want *his* code to be out in the wild. I don't think he would take great objection to a complete rewrite making it into the wild. It is unclear about his opinions about a hybrid code-base or how he would feel about naming it Alpha, but I'm sure someone can ask him.

Most of my coding is now for iOS and I don't know if a jump to the iPad would make a lot of sense for Alpha (I can't figure out exactly how I would redo the user interface to provide a pleasant touch UI for much of what I do in Alpha, but others may have some ideas). A move to iOS would require the UI  to be completely reworked since iOS relies on UIKit, not AppKit. Without a clear kernel/interface division of code, a move to iOS would probably be very difficult.

Personally, I would be more inclined to abandon the current code base and start relatively fresh. I haven't looked at the Alpha code in a few years, but a lot of what it struggled to do is part of the Cocoa API (I'm thinking of a lot of the file system interface code that I recoded when we moved it to Carbon and the text engine code that I was trying to rewrite using ASTUI). Keep in mind that Pete was trying to implement customized behaviors on Macintosh System 7 (6?) that are now considered standard behaviors (or out of date behaviors) on OS X. A new start would probably allow us to reduce the total code and maybe get a more modern UI experience.

There are also maintenance issues with the current code base. Starting with a smaller, cleaner, more-modular, more-testable skeleton might be easier to maintain (especially if it is going to be open source) than trying to continue to graft new pieces onto the old skeleton which was (and may still be) highly intertwined. When I was working on rewriting the syntax highlighting code (a project that never got completed), I remember touching source files all over the place and that type of coding is going to be a nightmare once you get more than a handful of programmers working on the project simultaneously.

Aaron

Le 27 juin 11 à 17:33, Lars Hellström a écrit :

> Bernard Desgraupes skrev 2011-06-27 15.21:
>> The idea is that AlphaCocoa will support all kinds of input or output
>> encodings (as AlpahTk does already). It should be able to edit text  
>> files in
>> almost any encoding (known to Tcl).
>> I think the internal buffer will be in UTF16.
>> I have also compiled the ICU library as a framework and embedded it  
>> inside
>> the application in order to have full Unicode support (Tcl is  
>> incomplete in
>> this regard).
>
> It woefully is.
>
> For data interchange (Tcl <-> full Unicode), you might find my half- 
> baked
> UTF package useful. Code is at http://wiki.tcl.tk/_repo/UTF/, and I've
> written something about it on http://wiki.tcl.tk/14026. But on the  
> other
> hand, ICU might provide the equivalent functionality as-is.


Thanks for the code. Do you have a Makefile or a Makefile.in (it is  
looked for by the ./configure script) ?

Bernard

Bernard Desgraupes skrev 2011-06-27 20.37:
>
> Le 27 juin 11 à 17:33, Lars Hellström a écrit :
>> For data interchange (Tcl<->  full Unicode), you might find my half-
>> baked
>> UTF package useful. Code is at http://wiki.tcl.tk/_repo/UTF/, and I've
>> written something about it on http://wiki.tcl.tk/14026. But on the
>> other
>> hand, ICU might provide the equivalent functionality as-is.
>
>
> Thanks for the code. Do you have a Makefile or a Makefile.in (it is
> looked for by the ./configure script) ?

As implied by what I wrote in the wiki page, the Makefile.in (and 
pkgIndex.tcl.in) were unmodified from the standard Tcl sampleextension.

/Lars

On Jun 27, 2011, at 11:57 PM, Eeyore <eeyore@...> wrote:

> I believe Pete's concern about open sourcing the code was that he didn't want *his* code to be out in the wild. I don't think he would take great objection to a complete rewrite making it into the wild. It is unclear about his opinions about a hybrid code-base or how he would feel about naming it Alpha, but I'm sure someone can ask him.

Pete was definitely opposed to open sourcing his code base (even though there isn't a lot of his code left). Not that it mattered, as Pete's vote was the only one that counted, but in some regards I wasn't a fan of open sourcing it either, simply because the code base was too entangled already and I worried that lots of people fiddling with things would only make things worse. That fear was probably unfounded, and the reality is that core has only been worked on by $\le 1$ programmer for many years now.

As far as going forward, AlphaTcl is already open source and Vince developed Alphatk with Pete's blessing as a completely independent project. Likewise, Pete sanctioned my splitting off some vestiges of his AppleEvent code in my open source TclAE project. IANAL, but I think that developing a new core from a pristine codebase to support AlphaTcl is completely kosher and the only thing Pete might have proprietary claim to is the name.

> Personally, I would be more inclined to abandon the current code base and start relatively fresh. I haven't looked at the Alpha code in a few years, but a lot of what it struggled to do is part of the Cocoa API (I'm thinking of a lot of the file system interface code that I recoded when we moved it to Carbon and the text engine code that I was trying to rewrite using ASTUI). Keep in mind that Pete was trying to implement customized behaviors on Macintosh System 7 (6?) that are now considered standard behaviors (or out of date behaviors) on OS X. A new start would probably allow us to reduce the total code and maybe get a more modern UI experience.
> 
> There are also maintenance issues with the current code base. Starting with a smaller, cleaner, more-modular, more-testable skeleton might be easier to maintain (especially if it is going to be open source) than trying to continue to graft new pieces onto the old skeleton which was (and may still be) highly intertwined. When I was working on rewriting the syntax highlighting code (a project that never got completed), I remember touching source files all over the place and that type of coding is going to be a nightmare once you get more than a handful of programmers working on the project simultaneously.

I completely concur with this assessment. The biggest things that slowed my efforts were the way that functionality was spread across numerous source files and my feeling that I needed to maintain backward compatibility, even though there were much better ways to do things in OS X.

I think a clean start on the core, with an eye on serving the most modern AlphaTcl, is the way forward. In fact, I would propose that this project start with refreshing the design of AlphaTcl. Strip away all obsolete core commands to leave only those we really want, and then add in those commands that can only be provided by the core that we wish we had. To some extent, this was always my approach to the core: provide the minimal functionality in C to allow our much larger community of Tcl programmers to do their work, but I think this is an opportunity to discard some really painful things that are only kept around for legacy reasons.

All that being said, I have no experience with Cocoa and I'm unlikely to have much time to contribute, but you never know.
>

> 3) Making good use of AlphaTk by relying as much as possible on Tcl/Tk seems
> to me to be a quite a good and possible fruitful idea, inasmuch this helps
> to minimise the recoding effort of the binary kernel (as described under
> point 2) above). Vince, nice to hear from you: Is really everything of
> AlphaTk in Tcl/Tk, or is there not some other "binary" kernel? Sorry, have
> never looked at the source code. If there is some binary kernel, what is the
> ratio?
>
> There is indeed a C kernel in AlphaTk, which Vince suggested he might open
> source. I don't know exactly which framework it is based on but I guess it
> is Carbon.

Actually there isn't really an obligatory C kernel in Alphatk. On
MacOS X I did make a couple of tiny changes to Tk (mostly just to
allow icons in the menubar which otherwise wasn't easy), but really
Alphatk is pure Tcl/Tk code. It makes use of quite a few 3rd party
libraries (for drag'n'drop, printing, etc - things that aren't yet
core to Tcl or Tk, so those are indeed compiled C code). But Alphatk
can be run without any C code at all.   Generally I used tclkit to
package everything up into a single executable for ease of use and
deployment.

In fact it is perfectly possible to think of developing a new
Alpha(Cocoa) by starting with Alphatk running on TkCocoa and then
replacing/augmenting it with new C/Obj-C coded pieces as desired.

I like Lars' comment that with Tcl's virtual filesystem capabilities,
an Alpha iPad app could indeed write to its own "filesystem" - e.g. by
writing a sqlite tcl-vfs module for iOS.

cheers,

Vince.

Hi Aaron,

nice to hear from you.



Le 27 juin 11 à 17:57, Eeyore a écrit :

> Before I was whisked away into the great oblivion that is my current  
> job about a decade ago, I did a little coding in the AlphaX core,  
> here are my thoughts on the move:
>
> Moving to Objective-C/Cocoa for the UI is probably the only way  
> forward, so this is generally a good move. See below for my thoughts  
> on the moving the core to Objective-C.
>
> I believe Pete's concern about open sourcing the code was that he  
> didn't want *his* code to be out in the wild. I don't think he would  
> take great objection to a complete rewrite making it into the wild.  
> It is unclear about his opinions about a hybrid code-base or how he  
> would feel about naming it Alpha, but I'm sure someone can ask him.

I agree, this is why the current code is not concerned here by the  
open sourcing proposal. An entirely new code base must be started in  
Cocoa.


> Personally, I would be more inclined to abandon the current code  
> base and start relatively fresh. I haven't looked at the Alpha code  
> in a few years, but a lot of what it struggled to do is part of the  
> Cocoa API (I'm thinking of a lot of the file system interface code  
> that I recoded when we moved it to Carbon and the text engine code  
> that I was trying to rewrite using ASTUI). Keep in mind that Pete  
> was trying to implement customized behaviors on Macintosh System 7  
> (6?) that are now considered standard behaviors (or out of date  
> behaviors) on OS X. A new start would probably allow us to reduce  
> the total code and maybe get a more modern UI experience.

That's right. The first parts I've started to code lead indeed to an  
amazing reduction of the code compared to what it would be in Carbon.


> There are also maintenance issues with the current code base.  
> Starting with a smaller, cleaner, more-modular, more-testable  
> skeleton might be easier to maintain (especially if it is going to  
> be open source) than trying to continue to graft new pieces onto the  
> old skeleton which was (and may still be) highly intertwined.

Object oriented programming indeed forces to adopt a clean modular way  
of doing things. The MVC model makes this even cleaner.


cheers,
Bernard

Hi Jon,


Le 29 juin 11 à 13:10, Jon a écrit :

> On Jun 27, 2011, at 11:57 PM, Eeyore <eeyore@...> wrote:
>
>> I believe Pete's concern about open sourcing the code was that he  
>> didn't want *his* code to be out in the wild. I don't think he  
>> would take great objection to a complete rewrite making it into the  
>> wild. It is unclear about his opinions about a hybrid code-base or  
>> how he would feel about naming it Alpha, but I'm sure someone can  
>> ask him.
>
> Pete was definitely opposed to open sourcing his code base (even  
> though there isn't a lot of his code left). Not that it mattered, as  
> Pete's vote was the only one that counted, but in some regards I  
> wasn't a fan of open sourcing it either, simply because the code  
> base was too entangled already and I worried that lots of people  
> fiddling with things would only make things worse. That fear was  
> probably unfounded, and the reality is that core has only been  
> worked on by $\le 1$ programmer for many years now.
>
> As far as going forward, AlphaTcl is already open source and Vince  
> developed Alphatk with Pete's blessing as a completely independent  
> project. Likewise, Pete sanctioned my splitting off some vestiges of  
> his AppleEvent code in my open source TclAE project. IANAL, but I  
> think that developing a new core from a pristine codebase to support  
> AlphaTcl is completely kosher and the only thing Pete might have  
> proprietary claim to is the name.

This is also my opinion concerning the new core built from scratch.  
None of the original code (what is left of it, namely the text engine)  
is reusable.
I think we should contact Pete regarding the use of the name: indeed  
being allowed to name the application just Alpha would be great.
Do you think you could do that, Jon or Vince ? In the name of all of us.

> I think a clean start on the core, with an eye on serving the most  
> modern AlphaTcl, is the way forward. In fact, I would propose that  
> this project start with refreshing the design of AlphaTcl. Strip  
> away all obsolete core commands to leave only those we really want,  
> and then add in those commands that can only be provided by the core  
> that we wish we had. To some extent, this was always my approach to  
> the core: provide the minimal functionality in C to allow our much  
> larger community of Tcl programmers to do their work, but I think  
> this is an opportunity to discard some really painful things that  
> are only kept around for legacy reasons.

I agree the AlphaTcl mechanisms must be entirely rethought (packages  
management, etc).

It would be great if we could define some sort of "public" API of Tcl  
procs to unify the definition of all the modes, menus and features.



> All that being said, I have no experience with Cocoa and I'm  
> unlikely to have much time to contribute, but you never know.

I don't have any experience either besides the fact that I've read  
Aaron Hillegass's "Cocoa Programming" three or four times in its  
second and its third editions. The documentation provided within Xcode  
gives an easy access to all the classes, methods, protocols etc. And  
there are tons of sample code all over the web.

I have the feeling there are things nevertheless which might be no so  
easy to implement in Cocoa. I'm currently working on redefining the  
menuRef and menuItem commands and have sometimes difficulties to find  
an equivalent in Cocoa to what was readily provided in the Carbon menu  
manager. Probably just a matter of getting used to the cocoa way.

So I think I'll now start a project on SourceForge (project name:  
alphacocoa ?) and install a wiki (trac?) there so that we can start  
entering RFEs and discuss the development choices.

> I agree the AlphaTcl mechanisms must be entirely rethought (packages
> management, etc).
> It would be great if we could define some sort of "public" API of Tcl procs
> to unify the definition of all the modes, menus and features.

I've always liked the Eclipse plugin mechanism - it is completely
general (based on some OSGi stuff), and relies on the simple idea that
any plugin can declare extension points (with lists of attributes)
that other plugins can then provide. This one mechanism is then used
for everything (it obviously supports versioning, etc), and has (in
Eclipse 3.x) scaled successfully to 1000s of packages. It is not that
far away from what AlphaTcl packages system tries to do, but is 1000x
more uniform, robust, well-designed, sensible, etc... It probably
wouldn't even be that much work to rip out what AlphaTcl uses and
replace it with such a mechanism.

> All that being said, I have no experience with Cocoa and I'm unlikely to
> have much time to contribute, but you never know.

That's why I really like the idea of a minimal set of core commands:
Tcl +Tk(?) + small set of custom stuff, and write everything else in
Tcl.

Vince.

Le 29 juin 11 à 15:38, Vince Darley a écrit :

>> I agree the AlphaTcl mechanisms must be entirely rethought (packages
>> management, etc).
>> It would be great if we could define some sort of "public" API of  
>> Tcl procs
>> to unify the definition of all the modes, menus and features.
>
> I've always liked the Eclipse plugin mechanism - it is completely
> general (based on some OSGi stuff), and relies on the simple idea that
> any plugin can declare extension points (with lists of attributes)
> that other plugins can then provide. This one mechanism is then used
> for everything (it obviously supports versioning, etc), and has (in
> Eclipse 3.x) scaled successfully to 1000s of packages. It is not that
> far away from what AlphaTcl packages system tries to do, but is 1000x
> more uniform, robust, well-designed, sensible, etc... It probably
> wouldn't even be that much work to rip out what AlphaTcl uses and
> replace it with such a mechanism.


It sounds good. Where is it documented. It's a long time I haven't  
worked with Eclipse.


Bernard

> It sounds good. Where is it documented. It's a long time I haven't worked
> with Eclipse.

Not sure of the best docs, but googling gives:

http://www.eclipse.org/articles/Article-Plug-in-architecture/plugin_architecture.html
http://en.wikipedia.org/wiki/Eclipse_(software)
http://en.wikipedia.org/wiki/Equinox_(OSGi)
http://aneeshkumarkb.blogspot.com/

which all seem fairly good.

Vince.

Le 27 juin 11 à 21:15, Lars Hellström a écrit :

> Bernard Desgraupes skrev 2011-06-27 20.37:
>>
>> Le 27 juin 11 à 17:33, Lars Hellström a écrit :
>>> For data interchange (Tcl<->  full Unicode), you might find my half-
>>> baked
>>> UTF package useful. Code is at http://wiki.tcl.tk/_repo/UTF/, and  
>>> I've
>>> written something about it on http://wiki.tcl.tk/14026. But on the
>>> other
>>> hand, ICU might provide the equivalent functionality as-is.
>>
>>
>> Thanks for the code. Do you have a Makefile or a Makefile.in (it is
>> looked for by the ./configure script) ?
>
> As implied by what I wrote in the wiki page, the Makefile.in (and
> pkgIndex.tcl.in) were unmodified from the standard Tcl  
> sampleextension.

Hi Lars,

I added my own Makefile.in and pkgIndex.tcl.in and it compiles out of  
the box.
I tested it from Alpha:

Welcome to AlphaX's AlphaTcl shell.
«» version
AlphaX 8.2rc3, Monday, 27 June 2011
«» package require UTF
0.1
«» lsearch -exact [encoding names] UTF-8
34
«» encoding convertfrom UTF-8 \303\266
ö
«» encoding convertto UTF-8 \u00F6
ö
etc...

Great! Can we ship it with the next release of AlphaX (8.2) ?

cheers,
Bernard

Bernard Desgraupes skrev 2011-06-30 10.33:
> Hi Lars,
>
> I added my own Makefile.in and pkgIndex.tcl.in and it compiles out of the box.
> I tested it from Alpha:
>
> Welcome to AlphaX's AlphaTcl shell.
> «» version
> AlphaX 8.2rc3, Monday, 27 June 2011
> «» package require UTF
> 0.1
> «» lsearch -exact [encoding names] UTF-8
> 34
> «» encoding convertfrom UTF-8 \303\266
> ö
> «» encoding convertto UTF-8 \u00F6
> ö
> etc...
>
> Great! Can we ship it with the next release of AlphaX (8.2) ?

"Hey, it compiles! Let's ship it!", eh?

Well, I'm flattered, and licensing-wise have no objection (consider the code 
BSD'ed), but I don't see that it would do much good for AlphaX. /Maybe/ it 
could be of use with Alphatk.

To clarify this, I suppose I'd better explain what the package does: It 
creates a Tcl encoding (programmatic rather than table-defined) named UTF-8. 
Tcl already comes with an utf-8 encoding built in, so how are the two 
different? Mainly in how they deal with four-byte UTF-8 sequences (i.e., 
characters outside the Basic Multilingual Plane from \u0000 to \uFFFF, which 
is the only range Tcl supports by default).

Tcl's standard utf-8 encoding decodes these, sees that they are outside the 
supported range, and substitutes a REPLACEMENT CHARACTER (don't recall 
anymore if it is \uFFFC or just ?). The UTF-8 encoding decodes these, sees 
that they are outside the supported range, and substitute the corresponding 
surrogate pair of characters (something like \uD800\DC00). It's not the same 
to anyone doing a [string index], but it can preserve the integrity of texts 
just happening to contain some non-BMP character. Conversely, when one does 
[encoding convertto UTF-8], a surrogate pair is converted to a four-byte 
sequences (rather than two three-byte sequences, as utf-8 would).

For an AlphaCocoa that treats its windows as containing a sequence of UTF-16 
strings, this UTF package behaviour makes perfect sense, as it is precisely 
what one wants to do when the file encoding is UTF-8. For AlphaX that is 
still restricted to macRoman in the text windows, I don't see how it would 
make a difference. In addition, I'm not entirely sure how stable this UTF 
package is: I think I had crashes when it was used as a channel encoding, 
and I can't recall whether I fixed those.

Finally, it does only provide part of the functionality I had envisioned. 
For one thing, I also wanted to provide UTF-16BE, UTF-16LE, and UTF-16 
(autodetection of Byte Order Mark) encodings; Tcl's built-in Unicode 
encoding is platform-dependently UTF-16BE or UTF-16LE, which is a bit 
unreliable. For another, I wanted to add all the #ifdef'ery needed to 
compile the package also for a Tcl that supports full Unicode. (IIRC, Tcl 
should work if compiled that way, even if Tk does not.)

Lars Hellström

PS: For AlphaCocoa, it may seem scary to have surrogate pairs in the buffer 
that are supposed to render as just one character, since that complicates 
indexing. However, Unicode forces us to face that that much earlier, due to 
the existence of combining accent characters -- isn't internationalisation 
fun? ;-) On the other hand, we may already have taken the basic steps 
towards decoupling column from character index, since we don't consider tabs 
(\t) to be just one column wide.

Le 30 juin 11 à 16:58, Lars Hellström a écrit :

> Bernard Desgraupes skrev 2011-06-30 10.33:
>> Hi Lars,
>>
>> I added my own Makefile.in and pkgIndex.tcl.in and it compiles out  
>> of the box.
>> I tested it from Alpha:
>>
>> Welcome to AlphaX's AlphaTcl shell.
>> «» version
>> AlphaX 8.2rc3, Monday, 27 June 2011
>> «» package require UTF
>> 0.1
>> «» lsearch -exact [encoding names] UTF-8
>> 34
>> «» encoding convertfrom UTF-8 \303\266
>> ö
>> «» encoding convertto UTF-8 \u00F6
>> ö
>> etc...
>>
>> Great! Can we ship it with the next release of AlphaX (8.2) ?
>
> "Hey, it compiles! Let's ship it!", eh?

Well, not only for this reason. It also passes the test suite.


> Well, I'm flattered, and licensing-wise have no objection (consider  
> the code BSD'ed), but I don't see that it would do much good for  
> AlphaX. /Maybe/ it could be of use with Alphatk.

The idea was just to make it available to people who have to deal with  
Unicode UTF16 and strings containing surrogate pairs.


> To clarify this, I suppose I'd better explain what the package does:  
> It creates a Tcl encoding (programmatic rather than table-defined)  
> named UTF-8. Tcl already comes with an utf-8 encoding built in, so  
> how are the two different? Mainly in how they deal with four-byte  
> UTF-8 sequences (i.e., characters outside the Basic Multilingual  
> Plane from \u0000 to \uFFFF, which is the only range Tcl supports by  
> default).
>
> Tcl's standard utf-8 encoding decodes these, sees that they are  
> outside the supported range, and substitutes a REPLACEMENT CHARACTER  
> (don't recall anymore if it is \uFFFC or just ?). The UTF-8 encoding  
> decodes these, sees that they are outside the supported range, and  
> substitute the corresponding surrogate pair of characters (something  
> like \uD800\DC00). It's not the same to anyone doing a [string  
> index], but it can preserve the integrity of texts just happening to  
> contain some non-BMP character. Conversely, when one does [encoding  
> convertto UTF-8], a surrogate pair is converted to a four-byte  
> sequences (rather than two three-byte sequences, as utf-8 would).

All this I had perfectly understood, this is why your extension seems  
to me useful, although I have been using my own implementation written  
in Tcl to convert surrogate pairs to and from  code-points in the SMP  
and beyond. The beauty of your solution is that it fits with the  
[encoding convertto/convertfrom] which makes it very easy to use.

>
> For an AlphaCocoa that treats its windows as containing a sequence  
> of UTF-16 strings, this UTF package behaviour makes perfect sense,  
> as it is precisely what one wants to do when the file encoding is  
> UTF-8. For AlphaX that is still restricted to macRoman in the text  
> windows, I don't see how it would make a difference. In addition,  
> I'm not entirely sure how stable this UTF package is: I think I had  
> crashes when it was used as a channel encoding, and I can't recall  
> whether I fixed those.
>
> Finally, it does only provide part of the functionality I had  
> envisioned. For one thing, I also wanted to provide UTF-16BE,  
> UTF-16LE, and UTF-16 (autodetection of Byte Order Mark) encodings;  
> Tcl's built-in Unicode encoding is platform-dependently UTF-16BE or  
> UTF-16LE, which is a bit unreliable. For another, I wanted to add  
> all the #ifdef'ery needed to compile the package also for a Tcl that  
> supports full Unicode. (IIRC, Tcl should work if compiled that way,  
> even if Tk does not.)

So just tell me when/if you want it to be embedded in AlphaX.

>
>
> PS: For AlphaCocoa, it may seem scary to have surrogate pairs in the  
> buffer that are supposed to render as just one character, since that  
> complicates indexing. However, Unicode forces us to face that that  
> much earlier, due to the existence of combining accent characters --  
> isn't internationalisation fun? ;-) On the other hand, we may  
> already have taken the basic steps towards decoupling column from  
> character index, since we don't consider tabs (\t) to be just one  
> column wide.
>

This is why I'll rely on the ICU library. All these complications  
(like getting a string length in a fully decomposed string etc.) are  
taken care of. I would not like to have to deal with them.

cheers,
Bernard