q-lang-users — general discussion, questions, suggestions, etc.

Re: [q-lang-users] Unicode

[John.Cowan <jc...@re...>]

Albert Graef scripsit:

> >John, is there a quick and dirty way to convert either a utf-8 multibyte 
> >char or a unicode point represented as a long int to a wchar_t? It 
> >appears that on Linux and Windows I should be able to simply cast the 
> >long value to wchar_t, but I guess that this breaks on other systems? I 
> >don't want to use iconv for that, to avoid the overhead, if there is a 
> >simpler way...
> 
> Hmm, taking another look at the glibc documentation, it seems that if 
> __STDC_ISO_10646__ is #define'd, then casting the unicode char number 
> (if it's below WCHAR_MAX) to wchar_t should work, no?

I don't actually know.  wchar_t is one of the badly underspecified parts of C:
ISO C only guarantees that sizeof(wchar_t) <= sizeof(char), yes, greater than
or equal to.  Windows makes it 16 bits, most Unix systems make it 32 bits.
I stay away from it.

One technique worth mentioning for implementing binary predicates about Unicode:
create a table of (start, stop+1) pairs covering the ranges for which the
predicate is true, and then binary-search the table.  If the answer is even,
the predicate is true, otherwise it's false.  For example, the predicate for
"is a Latin letter" is represented thus:

[0041, 005B, 0061, 007B, 00C0, 00D7, 00D8, 00F7, 00F8, 0220, 0222, 0234,
 1E00, 1E9C, 1EA0, 1EFA, FF21, FF3B, FF41, FF5B]

(This hasn't been updated to the very latest Unicode tables, and may miss some.)
If you unroll the binary search into a (hideously ugly, but provably correct)
tree of conditionals, the speed is pretty fast and the space almost nil.

-- 
John Cowan  jc...@re...  www.reutershealth.com  www.ccil.org/~cowan
If a traveler were informed that such a man [as Lord John Russell] was
leader of the House of Commons, he may well begin to comprehend how the
Egyptians worshiped an insect.  --Benjamin Disraeli

Re: [q-lang-users] Unicode

[Albert G. <Dr....@t-...>]

/me wrote:
> John, is there a quick and dirty way to convert either a utf-8 multibyte 
> char or a unicode point represented as a long int to a wchar_t? It 
> appears that on Linux and Windows I should be able to simply cast the 
> long value to wchar_t, but I guess that this breaks on other systems? I 
> don't want to use iconv for that, to avoid the overhead, if there is a 
> simpler way...

Hmm, taking another look at the glibc documentation, it seems that if 
__STDC_ISO_10646__ is #define'd, then casting the unicode char number 
(if it's below WCHAR_MAX) to wchar_t should work, no?

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Unicode

[Albert G. <Dr....@t-...>]

John.Cowan wrote:
> I think people who pick up a new programming language expect it to handle
> Unicode as the native type nowadays.  You have the opportunity to switch
> your basic string type to Unicode without getting huge complaints 
> about backward compatibility.  I urge you to take it.

Ok, I decided to give it a shot. An experimental new version of the 
interpreter is now in cvs which has native utf-8 encoded unicode 
strings. There's no automatic conversion of encodings yet, but if you're 
on a system with utf-8 as the system encoding (check your LC_TYPE) then 
most things should just work transparently, i.e., string operations 
treat utf-8 encoded multibyte chars as single characters. E.g. (warning: 
some Japanese characters ahead):

==> chars "abc 안녕하세요 \n"
["a","b","c"," ","안","녕","하","세","요"," ","\n"]

==> sub "abc 안녕하세요 \n" 4 6
"안녕하"

==> ["안".."앋"]
["안","앉","않","앋"]

==> "\0xc548"
"안"

There are some things which still need work, in particular, the isxxx 
char predicates and tolower/toupper in clib still need to be fixed. It 
would be nice if you could all try it and let me know how it works, bug 
reports are welcome. ;-)

John, is there a quick and dirty way to convert either a utf-8 multibyte 
char or a unicode point represented as a long int to a wchar_t? It 
appears that on Linux and Windows I should be able to simply cast the 
long value to wchar_t, but I guess that this breaks on other systems? I 
don't want to use iconv for that, to avoid the overhead, if there is a 
simpler way...

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Unicode

[Albert G. <Dr....@t-...>]

John.Cowan wrote:
> 	POSIX "ctype.h" knows but two cases, whereas Unicode knows
> 	three. In POSIX, only European Arabic digits can pass "isdigit",
> 	whereas Unicode has many sets of digits, all putatively equal. In
> 	POSIX "ctype.h", that which is "alnum" but not "alpha" must be a
> 	"digit", but Unicode is aware that not all numbers are digits,
> 	nor are all letters alphabetic. Unicode groks spacing and
> 	non-spacing marks, but POSIX comprehends them not.

Well, amendment 1 to ISO C90 has classification functions for wide 
characters (iswalpha and friends). These are available in glibc and 
AFAIK also on Windows, couldn't we just wrap these?

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Unicode

[Albert G. <Dr....@t-...>]

Albert Graef wrote:
> specialized set of operations to deal with UTF-8 strings on the 
> multibyte-char level (length calculation, indexing, slicing and 
> substring position should be all that is needed)

plus unicode character predicates and transformations, of course.

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Unicode

[Albert G. <Dr....@t-...>]

John.Cowan wrote:
> I think people who pick up a new programming language expect it to handle
> Unicode as the native type nowadays.  You have the opportunity to switch
> your basic string type to Unicode without getting huge complaints 
> about backward compatibility.  I urge you to take it.

Well, backward compatibility is only one consideration (no minor one for 
me, though, with the amount of Q code lying around on my harddisk ;-). 
Here are some others that come to my mind:

- Q (more precisely, clib) uses GNU regex for its regular expression 
stuff, which isn't unicode-aware either; this probably holds for glob, 
too. So there's the issue of legacy components which cannot be replaced 
easily.

- The performance issues with string processing; UTF-8 isn't ideal for 
stuff like char indexing.

- File positions are still measured in bytes, and using fseek() to 
reposition the file pointer is a sure way to break any on-the-fly 
conversion of the encoding.

- Formatted output via printf counts field widths in bytes, not wide 
characters. OTOH, wprintf cannot be used on a stream without breaking 
byte-oriented I/O (fread/fwrite).

So I'm not convinced that completely abstracting away the byte structure 
of strings is a good idea. What do others think? Would having a 
specialized set of operations to deal with UTF-8 strings on the 
multibyte-char level (length calculation, indexing, slicing and 
substring position should be all that is needed) be such a chore? Or 
would it be a reasonable compromise?

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Unicode

[John.Cowan <jc...@re...>]

Albert Graef scripsit:

> >It's important to be aware that the POSIX model of the isxxx predicates
> >is inadequate for Unicode.  Details on request.
> 
> I'm interested in the details, could you please elaborate?

Quoting myself from the Unicode FAQ:

	POSIX "ctype.h" knows but two cases, whereas Unicode knows
	three. In POSIX, only European Arabic digits can pass "isdigit",
	whereas Unicode has many sets of digits, all putatively equal. In
	POSIX "ctype.h", that which is "alnum" but not "alpha" must be a
	"digit", but Unicode is aware that not all numbers are digits,
	nor are all letters alphabetic. Unicode groks spacing and
	non-spacing marks, but POSIX comprehends them not.

IMHO the most important Unicode character categories are those lumped as
the General Category, which divides the entire codepoint space into
30 categories, themselves grouped into 7 supercategories (letter, number,
mark, punctuation, symbol, whitespace, other).

Relevant Unicode transformations are uppercasing, lowercasing, titlecasing,
and case folding, plus the four Unicode normalizations: decomposed, composed,
compatibility decomposed, and compatibility composed.

The numeric value of Unicode characters that are numbers is also significant.

See http://www.unicode.org/versions/Unicode4.0.0/ch04.pdf for information.
The ICU library (http://icu.sf.net) is the gold standard C/C++ implementation
library for everything Unicode, and I recommend it.  It's big, but it's
modularizable.

> Yes, that's ugly. The more I think about this the more I abhor the idea 
> to load the language itself with such complexities. Maybe it's better to 
> keep the language encoding-agnostic and push all unicode handling into 
> the library (just the way that it is done in C/C++, as opposed to Java/Tcl).

That's because C and C++ are stuck with the "character = byte" assumption
and have to build higher-level strings (unsigned short or long arrays,
typically).  You already have separate notions of "[character] string" and
"byte string".  So use byte strings for applications where you don't
care what the encoding is, and regular strings where you do.

> To these ends, clib would provide its own set of primitive operations 
> (say, u8length, u8chars, u8sub, etc.) for handling proper utf-8 encoded 
> strings. Then we could add a standard library module unicode.q on top of 
> that, with types UFile and UString and the corresponding operations, 
> which would handle the necessary conversions automatically and 
> transparently, as you suggested.

I think people who pick up a new programming language expect it to handle
Unicode as the native type nowadays.  You have the opportunity to switch
your basic string type to Unicode without getting huge complaints 
about backward compatibility.  I urge you to take it.

What do others think?

-- 
And through this revolting graveyard of the universe the muffled, maddening
beating of drums, and thin, monotonous whine of blasphemous flutes from
inconceivable, unlighted chambers beyond Time; the detestable pounding
and piping whereunto dance slowly, awkwardly, and absurdly the gigantic
tenebrous ultimate gods -- the blind, voiceless, mindless gargoyles whose soul
is Nyarlathotep. (Lovecraft) John Cowan|jc...@re...|ccil.org/~cowan

Re: [q-lang-users] Unicode

[Albert G. <Dr....@t-...>]

John.Cowan wrote:
> It's important to be aware that the POSIX model of the isxxx predicates
> is inadequate for Unicode.  Details on request.

I'm interested in the details, could you please elaborate?

> In addition, toupper and tolower have to operate at the string level,
> not merely the character level: they are not mere mappings in Unicode
> ("Maße" -> "MASSE", for one example).

That's no problem, as in Q these function already work on strings of 
arbitrary lengths anyway.

> The difficulty with this scheme is that you have to make the string-level
> operators work correctly, for some sense of "correctly", even with
> arbitrarily malformed UTF-8.

Yes, that's ugly. The more I think about this the more I abhor the idea 
to load the language itself with such complexities. Maybe it's better to 
keep the language encoding-agnostic and push all unicode handling into 
the library (just the way that it is done in C/C++, as opposed to Java/Tcl).

To these ends, clib would provide its own set of primitive operations 
(say, u8length, u8chars, u8sub, etc.) for handling proper utf-8 encoded 
strings. Then we could add a standard library module unicode.q on top of 
that, with types UFile and UString and the corresponding operations, 
which would handle the necessary conversions automatically and 
transparently, as you suggested.

The only actual change to the language itself would then be new escape 
sequences (\uXXXX) as shortcuts for utf-8 multibyte chars, and maybe a 
few related fixes in the string printing routine of the interpreter. Of 
course you could also use utf-8 encoded string literals in a source 
script (you already can, but they will print correctly only on a system 
which uses utf-8 as its native encoding).

What do you all think about this? I think that this might be the 
cleanest (if not most convenient) solution, also from a backward 
compatibility POV.

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] A small idea for better error checking

[Albert G. <Dr....@t-...>]

Peter Minten wrote:
> type StrictException : Exception = special const strict_exception A;

Yes, that's indeed the politically correct way of doing it, my code was 
correct but bit sloppy there. ;-)

> @-1
> marshal _ = error otherwise;

Yes, that's the right way to do it, if you want to supply a default rule 
that's supposed to be overridden by client modules. You might even 
consider to use a larger negative value (@-0x8000000 is the minimum 
iirc) just in case the client module plays its own priority tricks.

But note that this behaviour hasn't changed since priorities were 
introduced into the language, so I find it strange that the bug only 
surfaced with Q 6.2.

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] A small idea for better error checking

[Peter M. <pet...@wa...>]

Albert Graef wrote:
> Peter Minten wrote:
> 
>> So what strict does is throw an error when it receives an application.
> 
> 
> No need to go to C to achieve this, the following Q definition does more 
> or less the same:
> 
> strict (F X) = throw "oops";
> strict X = X otherwise;
> 
> (The only difference is that the exception is "oops" here, rather than 
> syserr 9.)
> 
>> Of course calling strict that way is rather irritating as it's likely to
>> be a much used function. Easier would be something like this:
>>
>> foo A B = bar A $ %baz B;
> 
> 
> How about something like:
> 
> public (strict) X @5;
> 

Thanks, I used:

"""
type StrictException : Exception = special const strict_exception A;

public (strict) X @5;
strict (F X) = throw $ strict_exception (F X);
strict X = X otherwise;
"""

Works perfectly.

>> PS: By the way, Smash was updated to version 0.2.1, fixed a small
>> priority bug that apparently was overseen by the compiler prior to 6.1 
>> and that destroyed the extensibility of marshal/unmarshal.
> 
> 
> Oops, what bug was that?

Well first something like:

marshal A:Array = ...;
marshal _ = error otherwise;
marshal D:Dict = ...;

Worked for a dict. Now I have to do

marshal A:Array = ...;
@-1
marshal _ = error otherwise;
@0
marshal D:Dict = ...;

To make the otherwise clause be considered after all the normal rules.

Greetings,

Peter Minten

Re: [q-lang-users] Unicode

[John.Cowan <jc...@re...>]

Albert Graef scripsit:

> - Fix the standard library functions chars and split, as well as the=20
> isxxx character predicates and toupper/tolower in clib.

It's important to be aware that the POSIX model of the isxxx predicates
is inadequate for Unicode.  Details on request.

In addition, toupper and tolower have to operate at the string level,
not merely the character level: they are not mere mappings in Unicode
("Ma=DFe" -> "MASSE", for one example).

> (2) Leave it up to the programmer. The interpreter just assumes that al=
l=20
> string data already is UTF-8 encoded and by itself doesn't touch the=20
> string data it reads/writes. When dealing with text data in other=20
> encodings, the programmer would have to use clib::iconv to do the=20
> conversion explicitly.

The difficulty with this scheme is that you have to make the string-level
operators work correctly, for some sense of "correctly", even with
arbitrarily malformed UTF-8.  If you do the mapping (which is really
quite cheap if amortized across a large object) you can guarantee
that you always have well-formed UTF-8 in the internals.

--=20
What asininity could I have uttered     John Cowan <jcowan@reutershealth.=
com>
that they applaud me thus?              http://www.reutershealth.com
        --Phocion, Greek orator         http://www.ccil.org/~cowan

Re: [q-lang-users] A small idea for better error checking

[Albert G. <Dr....@t-...>]

Peter Minten wrote:
> So what strict does is throw an error when it receives an application.
> This is useful for debugging. Take for example the following piece of code:
> 
> fred = ... ; //calls foo somewhere
> foo A B = bar A $ strict $ baz B;
> 
> Now say baz fails. Normally you'd get a pretty big unresolved
> expression. Now strict causes the program to crash with a message that
> baz B didn't resolve. Another option would be to throw an exception and 
> giving the program a chance to recover.

I should remark here that if an external function returns __ERROR, that 
actually raises an exception (syserr 9), which can be handled with catch 
as usual.

Also, to achieve the same effect, you don't need a specialized function 
like "strict" in your example, instead you could just use an "error 
rule" like:

baz B = ...; // real definition of baz
baz B = throw "baz goofed" otherwise; // error rule

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] A small idea for better error checking

[Albert G. <Dr....@t-...>]

Peter Minten wrote:
> So what strict does is throw an error when it receives an application.

No need to go to C to achieve this, the following Q definition does more 
or less the same:

strict (F X) = throw "oops";
strict X = X otherwise;

(The only difference is that the exception is "oops" here, rather than 
syserr 9.)

> Of course calling strict that way is rather irritating as it's likely to
> be a much used function. Easier would be something like this:
> 
> foo A B = bar A $ %baz B;

How about something like:

public (strict) X @5;

> PS: By the way, Smash was updated to version 0.2.1, fixed a small
> priority bug that apparently was overseen by the compiler prior to 6.1 
> and that destroyed the extensibility of marshal/unmarshal.

Oops, what bug was that?

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

[q-lang-users] A small idea for better error checking

[Peter M. <pet...@wa...>]

Hi all,

As most of us know Q's rewrite model doesn't allow for much error 
checking. However there is a simple trick for finding errors that might 
be used to make this a little better.

The trick is that when a reduction fails you get an Q value, which I
believe is called an application. Applications can also be given back on
purpose which is why they're not suited for general failure finding,
however the programmer can know if an application should be returned or
not. If one defines a simple function "strict" it should be possible to
catch applications. On the C level that can be done like this (leaving
out the obvious checks and improvements):

FUNCTION(strict,strict,argc,argv)
{
   expr a, b; //throwaway

   if(isapp(argv[0],&a,&b))
     // print out an error message somehow
     return __ERROR;
   else
     return argv[0];
}

So what strict does is throw an error when it receives an application.
This is useful for debugging. Take for example the following piece of code:

fred = ... ; //calls foo somewhere
foo A B = bar A $ strict $ baz B;

Now say baz fails. Normally you'd get a pretty big unresolved
expression. Now strict causes the program to crash with a message that
baz B didn't resolve. Another option would be to throw an exception and 
giving the program a chance to recover.

Of course calling strict that way is rather irritating as it's likely to
be a much used function. Easier would be something like this:

foo A B = bar A $ %baz B;

Function application beats any operator so the expression would be
something like: bar A (% (baz B)).

Just an idea :-).

Greetings,

Peter Minten

PS: By the way, Smash was updated to version 0.2.1, fixed a small
priority bug that apparently was overseen by the compiler prior to 6.1 
and that destroyed the extensibility of marshal/unmarshal.

Re: [q-lang-users] Unicode

[Albert G. <Dr....@t-...>]

Ok, I finally decided that it's time to do something about unicode 
support, so I read the relevant docs (thanks, John, for your pointers, 
they were quite useful).

John Cowan wrote:
> There will be five places where Unicode has to be addressed:  in pulling
> substrings out of strings, in reading, in writing, in converting from
> strings to byte strings, and in converting from byte strings to strings.
> In the last two cases, it is desirable (but not necessary) to provide
> a method of overriding the system standard external encoding such as
> Latin-1 which is generated or interpreted respectively.

AFAICS, here's what would be needed to get at least halfway-decent 
support for unicode/utf-8 in Q:

- Add UTF-8/multibyte character support to the interpreter. This 
affects, in particular, runtime string typing (since Char objects might 
consist of more than one byte) and marshalling (printing string objects 
in the interpreter), as well as the builtins (#), (!), sub, substr, pos, 
ord, char, succ, pred, enum.

- Fix the standard library functions chars and split, as well as the 
isxxx character predicates and toupper/tolower in clib.

- Add the usual localization stuff to clib: 
setlocale/localeconv/nl_langinfo, strfmon, strcoll/strxfrm, iconv, 
gettext and friends.

That should be all that is needed to have unicode just working when 
running on a system which has UTF-8 as the default encoding. But, as 
John pointed out, on systems using a different encoding there is still 
the issue of converting strings passed to or obtained from the system 
(including string constants in the source script and on the command 
line). There are basically two ways to deal with this:

(1) Add automatic conversions to all operations which read/write strings 
from/to the system and byte string data. This is what John proposed, and 
is certainly the most convenient for the programmer. But is this really 
desirable? Doing the conversions automatically means that you always 
have to pay for it, even if you use fget to slurp in big 7 bit ascii 
files. And if you read or write a file which happens to be in an 
encoding different from the system encoding then the builtin conversion 
would garble the string data.

(2) Leave it up to the programmer. The interpreter just assumes that all 
string data already is UTF-8 encoded and by itself doesn't touch the 
string data it reads/writes. When dealing with text data in other 
encodings, the programmer would have to use clib::iconv to do the 
conversion explicitly.

I'm actually leaning towards solution (2) since it gives greater freedom 
to the programmer and avoids the conversion overhead when it's not 
needed. Also, it is more in line with the current implementation which 
doesn't mangle string data behind the scenes either. And it has the 
added benefit that scripts containing string constants with extended 
characters would be portable across platforms (which is not the case if 
you implicitly assume the system encoding, so that scripts always have 
to be written in the local encoding).

Opinions?

Cheers,
Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] qcwrap

[Albert G. <Dr....@t-...>]

se...@ge... wrote:
> qcwrap consumes this with ease, and the dynamic version compiles and runs
> beautifully.  The statically compiled version produced by:
> 
>   gcc -o fact_static fact.c -static -lqint -lq -lgmp -lpthread -lm -ldl
> 
> segfaults when I run it.

Yes, this is a bug. You could try to rebuild libqint after enabling the 
call to LTDL_SET_PRELOADED_SYMBOLS in libmain() (q.c, line 2211), but 
then you might have to link your qcwrapped program using libtool. If you 
try that please let me know how it works.

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

[q-lang-users] qcwrap

[<se...@ge...>]

First, thank you for the responses, Herr Dr. Graef.

I've been playing with qcwrap, and am having a couple of problems.  Here's
a little Q script that I'm compiling:

  #!/usr/bin/env q
  #! -cmain

  fact N = subfact N 1;
  subfact N A = subfact (N-1) (N*A) if N > 0;
              = -1                  if N < 0;
              = A                   otherwise;

  main = writes $ (str $ fact $ val $ ARGS!1) ++ "\n";

I was first working with a script that generated a bitset of a given
number, but it ran too quickly to be useful in timing comparisons.

A sample run of this:

  $ ./fact.q 5
  120
  ()

qcwrap consumes this with ease, and the dynamic version compiles and runs
beautifully.  The statically compiled version produced by:

  gcc -o fact_static fact.c -static -lqint -lq -lgmp -lpthread -lm -ldl

segfaults when I run it.  This may not be related to Q, but rather my
compilation options -- I don't statically compile programs much, and need
to make sure I'm calling gcc properly.

Not too surprisingly, compiling doesn't improve execution times much,
although for very short scripts, the overhead of loading the Q interpreter
is much more of a factor:

  ./fact_dynamic     5  0.00s user 0.01s system 60% cpu 0.017 total
  ./fact.q           5  0.05s user 0.01s system 24% cpu 0.256 total
  ./fact_dynamic 50005  4.12s user 0.06s system 93% cpu 4.495 total
  ./fact.q       50005  4.11s user 0.08s system 72% cpu 5.757 total

--- SER

Re: [q-lang-users] Executables and modules

[Albert G. <Dr....@t-...>]

se...@ge... wrote:
> 64-bit support is, for me, rather important.

I hope that I can fix this during summer. I need to, because I'm giving 
a Q course in autumn, and one machines in our lab now has a 64 bit cpu.

> I really liked Haskell, until I hit the lazy-IO/monad hill, after which
> every non-trivial project seems like a chore.

Yes, monads look very nice on paper, but in practice they are a major 
*%&!. The awkward squad continues to strive ... ;-)

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Executables and modules

[Albert G. <Dr....@t-...>]

se...@ge... wrote:
> I agree.  Compiling to an intermediate language, such as Objective-C,
> would be the way to go.

Or use the gcc backend directly. That gives you more control about code 
generation and optimization, and having to go through a C compile stage 
slows things down quite a bit.

> I didn't mean "dependency" as "another bothersome thing to compile"; I
> meant "dependancy" as in "something else to break".

Ah ok, I see. You certainly have a point there.

> Yes; I hadn't seen qcwrap.  If the executables didn't link to libq or
> libqint -- now, there's a thought... I suppose I could just statically
> link to libq and libqint; then I'd only have to worry about peripheral
> libraries.  Is there a way to make qcwrap "pull in" all .q dependencies
> for an application?

qcwrap doesn't wrap the source script, it wraps the byte code, so you 
don't have to worry about .q dependencies when using it. Instead of 
statically linking to libq/libqint, you could also ship the shared libs 
with your application and rely on library versions. Hmm, that reminds me 
that I forgot to increment the libqint version in Q 6.1/6.2, I'll have 
to fix that in the next release... ;-)

> Ouch.  Yeah, I forgot that.  Hrm.  I've always liked type inferrence, and
> have often wondered why dynamically typed anguages couldn't implement type
> inferrence based on duck typing.  Especially in the case of Q, which does
> perform a compilation step.  But, this is Yet Another Feature.

Well, the amount of ad-hoc polymorphism in Q is essentially unlimited, 
and you can't decide statically whether a given expression is able to 
"respond" to a given "method" (function). That means that "duck typing" 
becomes undecidable.

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Executables and modules

[<se...@ge...>]

> Sean E. Russell wrote:
...
>> Gräf, you chose to compile to bytecode; have you considered compiling to
>> native code, or to some other intermediate language that could be
...
> Yes, of course I have considered this, but there are some things in the
> language, especially the runtime special forms, which makes this
> difficult and I can't say what the actual speed improvements would be.

Those are hard to tell;  since Q's performance is similar to interpreted
Haskell and Q is, itself, similar to Haskell, you might expect similar
performance gains as one sees between ghci and ghc.  The extrapolation may
be entirely incorrect, but one can hope.  Furthermore, as I said earlier,
one would hope that since Q has eager evaluation, it may actually exceed
compiled Haskell, which would be nice.

> The key here is to find an existing backend (maybe gcc) which already

I agree.  Compiling to an intermediate language, such as Objective-C,
would be the way to go.

>> another dependency failure point.  That is, yet one more dependency that
...
> Well, leaving endianness and 64 bit issues aside (which would be pretty
> much the same, whether you have a bytecode compiler or a native compiler
> with associated runtime), the core of an interpreter is usually very
> portable anyway. In my experience, portability issues almost always

I didn't mean "dependency" as "another bothersome thing to compile"; I
meant "dependancy" as in "something else to break".

If I have an application that *only* links against libc, then my
application will only break if libc breaks compatibility.  The more things
I link against, the more risk I have of my application breaking when I
upgrade my system.  This is a maintenance issue, not an installation
issue.  I want to be able to write a program that does a job, and not have
to worry about it suddenly not working just because I upgraded my
installation of Q.

This problem isn't entirely avoidable; there are always dependencies. 
However, as I reduce the number of dependencies, the reliability of my
applications improves.  In my experience, interpreters are a
disproportionately large source of incompatabilities.

Interpreters are great for development, but nothing beats native
compilation for deployment.

> Concerning the 2nd point (ability to make executables): We already have
> that, it's called qcwrap.

Yes; I hadn't seen qcwrap.  If the executables didn't link to libq or
libqint -- now, there's a thought... I suppose I could just statically
link to libq and libqint; then I'd only have to worry about peripheral
libraries.  Is there a way to make qcwrap "pull in" all .q dependencies
for an application?

> Don't forget static vs. dynamic typing. That's what gives both ML and

Ouch.  Yeah, I forgot that.  Hrm.  I've always liked type inferrence, and
have often wondered why dynamically typed anguages couldn't implement type
inferrence based on duck typing.  Especially in the case of Q, which does
perform a compilation step.  But, this is Yet Another Feature.

Thanks for the response.

Tschuss.

--- SER

Re: [q-lang-users] Executables and modules

[<se...@ge...>]

> Have you taken a look at the "core" page? Quite a few modules are
> already included in the core. Together with the addons page, this lists
> everything that is available right now.

Ok, that's what I wanted to know.

> SQLite looks nice, because it doesn't need any setup, which can be

Yes; I like it primarily because it doesn't require a *server*, per se. 
It is an embedded database much like BerkeleyDB or gdb, but with SQL
support.  In any case, as I said, I wasn't asking _specifically_ about
SQLite -- I was just curious if there was another centralized list of
bindings.

> See, there's only so much that a single developer -- myself, with

I wasn't criticizing Q for a lack of bindings :-)  I was simply curious
whether there was another resource I should be trolling when looking for
libraries, before I start writing my own bindings.

> finished, I want to work on Q 6.3 which will have 64 bit and unicode
> support. So yes, I could use some help with the modules. ;-)

64-bit support is, for me, rather important.  I'm playing with Q at work
when I can justify it, but until it has 64-bit support, I can't use it at
home :-(.  Anyway, I'm not pressuring you; I have my own open source
projects with people clamoring for features that I don't have time to
write, so I know how it goes.

Incidentally, I wanted to mention that -- so far -- I really like Q.  I
really liked Haskell, until I hit the lazy-IO/monad hill, after which
every non-trivial project seems like a chore.  So, thanks for Q.  It is
keeping me busy :-)

--- SER

Re: [q-lang-users] Executables and modules

[Albert G. <Dr....@t-...>]

Sean E. Russell wrote:
> 1) You knew some annoying person was bound to ask this, sooner or later: Dr. 
> Gräf, you chose to compile to bytecode; have you considered compiling to 
> native code, or to some other intermediate language that could be compiled to 
> native code?  

Yes, of course I have considered this, but there are some things in the 
language, especially the runtime special forms, which makes this 
difficult and I can't say what the actual speed improvements would be. 
The key here is to find an existing backend (maybe gcc) which already 
supports a lot of architectures and is suitable to handle the special 
requirements of Q; otherwise I'd end up having to support all those 
different architectures myself, a nightmare. That's one of the key 
points which speaks in favour of an interpreter: it's much easier to 
make it portable, at least nowadays where a decent C compiler is 
available for virtually every platform under the sun (no pun intended ;-).

> 	In addition to the two common reasons for wanting this -- speed and the 
> ability to make executables -- there is a third reason: interpreters add 
> another dependency failure point.  That is, yet one more dependency that 
> could change and cause an application to break.  In particular, interpreters 
> are often more brittle than other dependencies (like libc), since they tend 
> to change more radically and more often.

Well, leaving endianness and 64 bit issues aside (which would be pretty 
much the same, whether you have a bytecode compiler or a native compiler 
with associated runtime), the core of an interpreter is usually very 
portable anyway. In my experience, portability issues almost always 
arise in the addon libraries. (At least that's true now that we have C 
compilers+libraries adhering to a standard everywhere. It used to be a 
lot different in the "bad old days.")

Concerning the 2nd point (ability to make executables): We already have 
that, it's called qcwrap.

> 2) Exactly why is Q's speed so similar to Haskell's?  The consensus appears to 
> be that Haskell's lazy evaluation is (by and large) what kills it in 
> performance comparisons.  Q, however, is by default eager.  The interpreted 
> nature of Q (and evaluated Haskell) probably masks issues such as lazy/eager 
> performance issues, but any thoughts on this are welcome.

Don't forget static vs. dynamic typing. That's what gives both ML and 
Haskell a big edge in computation speed over more dynamic languages. 
Concerning lazy vs. eager, the performance issues with the former only 
become apparent when large intermediate terms have to be constructed. 
With simple tail-recursive definitions, even interpreted Haskell can be 
very fast, and then it may well be that lazy but unboxed evaluation is 
faster than an eager but boxed computation. OTOH, I found that with 
plain heavy-duty recursion on simple objects, such as the tak benchmark, 
the Q interpreter indeed is faster than Hugs; so it all depends on the 
type of benchmark.

Cheers,
Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Executables and modules

[Albert G. <Dr....@t-...>]

Sean E. Russell wrote:
> 3) Is the addons page a comprehensive source of libraries for Q?  For example, 
> one of the first things I look for a library for any language I research is a 
> binding to SQLite, since I use it so much in a variety of projects.  Note 
> that I'm not *specifically* asking about SQLite bindings; I'm asking whether 
> there's an "RAA" (Ruby Application Archive, 
> http://raa.ruby-lang.org/index.html) for Q.

Have you taken a look at the "core" page? Quite a few modules are 
already included in the core. Together with the addons page, this lists 
everything that is available right now.

SQLite looks nice, because it doesn't need any setup, which can be 
awkward with other databases. If the C API is simple enough, you could 
probably write a SWIG wrapper for this in an afternoon. For the time 
being, there's an ODBC module in the core.

See, there's only so much that a single developer -- myself, with 
occasional help from my interns -- can do. We've been very busy lately 
bringing Q's multimedia library to a state where it compares quite 
favourably to anything else available in the FPL world (and even to some 
traditional scripting languages). Now that this is more or less 
finished, I want to work on Q 6.3 which will have 64 bit and unicode 
support. So yes, I could use some help with the modules. ;-)

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

[q-lang-users] ANN: Q-OpenAL 1.0 and Q-OpenGL 1.1

[Albert G. <Dr....@t-...>]

Hi all,

I've just released a new module which lets you interface to OpenAL 
(http://www.openal.org). Thanks to Jonas Joebges for his work on this. A 
bugfix release of the Q-OpenGL module is also available.

Enjoy :)
Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

[q-lang-users] Executables and modules

[Sean E. R. <se...@ge...>]

Hello again,

I have a few more questions that haven't been asked in the mailing list yet.

1) You knew some annoying person was bound to ask this, sooner or later: Dr=
=2E=20
Gr=C3=A4f, you chose to compile to bytecode; have you considered compiling =
to=20
native code, or to some other intermediate language that could be compiled =
to=20
native code? =20
	In addition to the two common reasons for wanting this -- speed and the=20
ability to make executables -- there is a third reason: interpreters add=20
another dependency failure point.  That is, yet one more dependency that=20
could change and cause an application to break.  In particular, interpreter=
s=20
are often more brittle than other dependencies (like libc), since they tend=
=20
to change more radically and more often.

2) Exactly why is Q's speed so similar to Haskell's?  The consensus appears=
 to=20
be that Haskell's lazy evaluation is (by and large) what kills it in=20
performance comparisons.  Q, however, is by default eager.  The interpreted=
=20
nature of Q (and evaluated Haskell) probably masks issues such as lazy/eage=
r=20
performance issues, but any thoughts on this are welcome.

3) Is the addons page a comprehensive source of libraries for Q?  For examp=
le,=20
one of the first things I look for a library for any language I research is=
 a=20
binding to SQLite, since I use it so much in a variety of projects.  Note=20
that I'm not *specifically* asking about SQLite bindings; I'm asking whethe=
r=20
there's an "RAA" (Ruby Application Archive,=20
http://raa.ruby-lang.org/index.html) for Q.

Thank you.

=2D-=20
=2D-- SER

"As democracy is perfected, the office of president represents,=20
more and more closely, the inner soul of the people.  On some=20
great and glorious day the plain folks of the land will reach=20
their heart's desire at last and the White House will be adorned=20
by a downright moron."        -  H.L. Mencken (1880 - 1956)