q-lang-users

[q-lang-users] Executables and modules

[Sean E. R. <se...@ge...>]

Hello again,

I have a few more questions that haven't been asked in the mailing list yet.

1) You knew some annoying person was bound to ask this, sooner or later: Dr=
=2E=20
Gr=C3=A4f, you chose to compile to bytecode; have you considered compiling =
to=20
native code, or to some other intermediate language that could be compiled =
to=20
native code? =20
	In addition to the two common reasons for wanting this -- speed and the=20
ability to make executables -- there is a third reason: interpreters add=20
another dependency failure point.  That is, yet one more dependency that=20
could change and cause an application to break.  In particular, interpreter=
s=20
are often more brittle than other dependencies (like libc), since they tend=
=20
to change more radically and more often.

2) Exactly why is Q's speed so similar to Haskell's?  The consensus appears=
 to=20
be that Haskell's lazy evaluation is (by and large) what kills it in=20
performance comparisons.  Q, however, is by default eager.  The interpreted=
=20
nature of Q (and evaluated Haskell) probably masks issues such as lazy/eage=
r=20
performance issues, but any thoughts on this are welcome.

3) Is the addons page a comprehensive source of libraries for Q?  For examp=
le,=20
one of the first things I look for a library for any language I research is=
 a=20
binding to SQLite, since I use it so much in a variety of projects.  Note=20
that I'm not *specifically* asking about SQLite bindings; I'm asking whethe=
r=20
there's an "RAA" (Ruby Application Archive,=20
http://raa.ruby-lang.org/index.html) for Q.

Thank you.

=2D-=20
=2D-- SER

"As democracy is perfected, the office of president represents,=20
more and more closely, the inner soul of the people.  On some=20
great and glorious day the plain folks of the land will reach=20
their heart's desire at last and the White House will be adorned=20
by a downright moron."        -  H.L. Mencken (1880 - 1956)

Re: [q-lang-users] Executables and modules

[Albert G. <Dr....@t-...>]

Sean E. Russell wrote:
> 3) Is the addons page a comprehensive source of libraries for Q?  For example, 
> one of the first things I look for a library for any language I research is a 
> binding to SQLite, since I use it so much in a variety of projects.  Note 
> that I'm not *specifically* asking about SQLite bindings; I'm asking whether 
> there's an "RAA" (Ruby Application Archive, 
> http://raa.ruby-lang.org/index.html) for Q.

Have you taken a look at the "core" page? Quite a few modules are 
already included in the core. Together with the addons page, this lists 
everything that is available right now.

SQLite looks nice, because it doesn't need any setup, which can be 
awkward with other databases. If the C API is simple enough, you could 
probably write a SWIG wrapper for this in an afternoon. For the time 
being, there's an ODBC module in the core.

See, there's only so much that a single developer -- myself, with 
occasional help from my interns -- can do. We've been very busy lately 
bringing Q's multimedia library to a state where it compares quite 
favourably to anything else available in the FPL world (and even to some 
traditional scripting languages). Now that this is more or less 
finished, I want to work on Q 6.3 which will have 64 bit and unicode 
support. So yes, I could use some help with the modules. ;-)

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Executables and modules

[<se...@ge...>]

> Have you taken a look at the "core" page? Quite a few modules are
> already included in the core. Together with the addons page, this lists
> everything that is available right now.

Ok, that's what I wanted to know.

> SQLite looks nice, because it doesn't need any setup, which can be

Yes; I like it primarily because it doesn't require a *server*, per se. 
It is an embedded database much like BerkeleyDB or gdb, but with SQL
support.  In any case, as I said, I wasn't asking _specifically_ about
SQLite -- I was just curious if there was another centralized list of
bindings.

> See, there's only so much that a single developer -- myself, with

I wasn't criticizing Q for a lack of bindings :-)  I was simply curious
whether there was another resource I should be trolling when looking for
libraries, before I start writing my own bindings.

> finished, I want to work on Q 6.3 which will have 64 bit and unicode
> support. So yes, I could use some help with the modules. ;-)

64-bit support is, for me, rather important.  I'm playing with Q at work
when I can justify it, but until it has 64-bit support, I can't use it at
home :-(.  Anyway, I'm not pressuring you; I have my own open source
projects with people clamoring for features that I don't have time to
write, so I know how it goes.

Incidentally, I wanted to mention that -- so far -- I really like Q.  I
really liked Haskell, until I hit the lazy-IO/monad hill, after which
every non-trivial project seems like a chore.  So, thanks for Q.  It is
keeping me busy :-)

--- SER

Re: [q-lang-users] Executables and modules

[Albert G. <Dr....@t-...>]

se...@ge... wrote:
> 64-bit support is, for me, rather important.

I hope that I can fix this during summer. I need to, because I'm giving 
a Q course in autumn, and one machines in our lab now has a 64 bit cpu.

> I really liked Haskell, until I hit the lazy-IO/monad hill, after which
> every non-trivial project seems like a chore.

Yes, monads look very nice on paper, but in practice they are a major 
*%&!. The awkward squad continues to strive ... ;-)

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Executables and modules

[Albert G. <Dr....@t-...>]

Sean E. Russell wrote:
> 1) You knew some annoying person was bound to ask this, sooner or later: Dr. 
> Gräf, you chose to compile to bytecode; have you considered compiling to 
> native code, or to some other intermediate language that could be compiled to 
> native code?  

Yes, of course I have considered this, but there are some things in the 
language, especially the runtime special forms, which makes this 
difficult and I can't say what the actual speed improvements would be. 
The key here is to find an existing backend (maybe gcc) which already 
supports a lot of architectures and is suitable to handle the special 
requirements of Q; otherwise I'd end up having to support all those 
different architectures myself, a nightmare. That's one of the key 
points which speaks in favour of an interpreter: it's much easier to 
make it portable, at least nowadays where a decent C compiler is 
available for virtually every platform under the sun (no pun intended ;-).

> 	In addition to the two common reasons for wanting this -- speed and the 
> ability to make executables -- there is a third reason: interpreters add 
> another dependency failure point.  That is, yet one more dependency that 
> could change and cause an application to break.  In particular, interpreters 
> are often more brittle than other dependencies (like libc), since they tend 
> to change more radically and more often.

Well, leaving endianness and 64 bit issues aside (which would be pretty 
much the same, whether you have a bytecode compiler or a native compiler 
with associated runtime), the core of an interpreter is usually very 
portable anyway. In my experience, portability issues almost always 
arise in the addon libraries. (At least that's true now that we have C 
compilers+libraries adhering to a standard everywhere. It used to be a 
lot different in the "bad old days.")

Concerning the 2nd point (ability to make executables): We already have 
that, it's called qcwrap.

> 2) Exactly why is Q's speed so similar to Haskell's?  The consensus appears to 
> be that Haskell's lazy evaluation is (by and large) what kills it in 
> performance comparisons.  Q, however, is by default eager.  The interpreted 
> nature of Q (and evaluated Haskell) probably masks issues such as lazy/eager 
> performance issues, but any thoughts on this are welcome.

Don't forget static vs. dynamic typing. That's what gives both ML and 
Haskell a big edge in computation speed over more dynamic languages. 
Concerning lazy vs. eager, the performance issues with the former only 
become apparent when large intermediate terms have to be constructed. 
With simple tail-recursive definitions, even interpreted Haskell can be 
very fast, and then it may well be that lazy but unboxed evaluation is 
faster than an eager but boxed computation. OTOH, I found that with 
plain heavy-duty recursion on simple objects, such as the tak benchmark, 
the Q interpreter indeed is faster than Hugs; so it all depends on the 
type of benchmark.

Cheers,
Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

Re: [q-lang-users] Executables and modules

[<se...@ge...>]

> Sean E. Russell wrote:
...
>> Gräf, you chose to compile to bytecode; have you considered compiling to
>> native code, or to some other intermediate language that could be
...
> Yes, of course I have considered this, but there are some things in the
> language, especially the runtime special forms, which makes this
> difficult and I can't say what the actual speed improvements would be.

Those are hard to tell;  since Q's performance is similar to interpreted
Haskell and Q is, itself, similar to Haskell, you might expect similar
performance gains as one sees between ghci and ghc.  The extrapolation may
be entirely incorrect, but one can hope.  Furthermore, as I said earlier,
one would hope that since Q has eager evaluation, it may actually exceed
compiled Haskell, which would be nice.

> The key here is to find an existing backend (maybe gcc) which already

I agree.  Compiling to an intermediate language, such as Objective-C,
would be the way to go.

>> another dependency failure point.  That is, yet one more dependency that
...
> Well, leaving endianness and 64 bit issues aside (which would be pretty
> much the same, whether you have a bytecode compiler or a native compiler
> with associated runtime), the core of an interpreter is usually very
> portable anyway. In my experience, portability issues almost always

I didn't mean "dependency" as "another bothersome thing to compile"; I
meant "dependancy" as in "something else to break".

If I have an application that *only* links against libc, then my
application will only break if libc breaks compatibility.  The more things
I link against, the more risk I have of my application breaking when I
upgrade my system.  This is a maintenance issue, not an installation
issue.  I want to be able to write a program that does a job, and not have
to worry about it suddenly not working just because I upgraded my
installation of Q.

This problem isn't entirely avoidable; there are always dependencies. 
However, as I reduce the number of dependencies, the reliability of my
applications improves.  In my experience, interpreters are a
disproportionately large source of incompatabilities.

Interpreters are great for development, but nothing beats native
compilation for deployment.

> Concerning the 2nd point (ability to make executables): We already have
> that, it's called qcwrap.

Yes; I hadn't seen qcwrap.  If the executables didn't link to libq or
libqint -- now, there's a thought... I suppose I could just statically
link to libq and libqint; then I'd only have to worry about peripheral
libraries.  Is there a way to make qcwrap "pull in" all .q dependencies
for an application?

> Don't forget static vs. dynamic typing. That's what gives both ML and

Ouch.  Yeah, I forgot that.  Hrm.  I've always liked type inferrence, and
have often wondered why dynamically typed anguages couldn't implement type
inferrence based on duck typing.  Especially in the case of Q, which does
perform a compilation step.  But, this is Yet Another Feature.

Thanks for the response.

Tschuss.

--- SER

Re: [q-lang-users] Executables and modules

[Albert G. <Dr....@t-...>]

se...@ge... wrote:
> I agree.  Compiling to an intermediate language, such as Objective-C,
> would be the way to go.

Or use the gcc backend directly. That gives you more control about code 
generation and optimization, and having to go through a C compile stage 
slows things down quite a bit.

> I didn't mean "dependency" as "another bothersome thing to compile"; I
> meant "dependancy" as in "something else to break".

Ah ok, I see. You certainly have a point there.

> Yes; I hadn't seen qcwrap.  If the executables didn't link to libq or
> libqint -- now, there's a thought... I suppose I could just statically
> link to libq and libqint; then I'd only have to worry about peripheral
> libraries.  Is there a way to make qcwrap "pull in" all .q dependencies
> for an application?

qcwrap doesn't wrap the source script, it wraps the byte code, so you 
don't have to worry about .q dependencies when using it. Instead of 
statically linking to libq/libqint, you could also ship the shared libs 
with your application and rely on library versions. Hmm, that reminds me 
that I forgot to increment the libqint version in Q 6.1/6.2, I'll have 
to fix that in the next release... ;-)

> Ouch.  Yeah, I forgot that.  Hrm.  I've always liked type inferrence, and
> have often wondered why dynamically typed anguages couldn't implement type
> inferrence based on duck typing.  Especially in the case of Q, which does
> perform a compilation step.  But, this is Yet Another Feature.

Well, the amount of ad-hoc polymorphism in Q is essentially unlimited, 
and you can't decide statically whether a given expression is able to 
"respond" to a given "method" (function). That means that "duck typing" 
becomes undecidable.

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag

[q-lang-users] qcwrap

[<se...@ge...>]

First, thank you for the responses, Herr Dr. Graef.

I've been playing with qcwrap, and am having a couple of problems.  Here's
a little Q script that I'm compiling:

  #!/usr/bin/env q
  #! -cmain

  fact N = subfact N 1;
  subfact N A = subfact (N-1) (N*A) if N > 0;
              = -1                  if N < 0;
              = A                   otherwise;

  main = writes $ (str $ fact $ val $ ARGS!1) ++ "\n";

I was first working with a script that generated a bitset of a given
number, but it ran too quickly to be useful in timing comparisons.

A sample run of this:

  $ ./fact.q 5
  120
  ()

qcwrap consumes this with ease, and the dynamic version compiles and runs
beautifully.  The statically compiled version produced by:

  gcc -o fact_static fact.c -static -lqint -lq -lgmp -lpthread -lm -ldl

segfaults when I run it.  This may not be related to Q, but rather my
compilation options -- I don't statically compile programs much, and need
to make sure I'm calling gcc properly.

Not too surprisingly, compiling doesn't improve execution times much,
although for very short scripts, the overhead of loading the Q interpreter
is much more of a factor:

  ./fact_dynamic     5  0.00s user 0.01s system 60% cpu 0.017 total
  ./fact.q           5  0.05s user 0.01s system 24% cpu 0.256 total
  ./fact_dynamic 50005  4.12s user 0.06s system 93% cpu 4.495 total
  ./fact.q       50005  4.11s user 0.08s system 72% cpu 5.757 total

--- SER

Re: [q-lang-users] qcwrap

[Albert G. <Dr....@t-...>]

se...@ge... wrote:
> qcwrap consumes this with ease, and the dynamic version compiles and runs
> beautifully.  The statically compiled version produced by:
> 
>   gcc -o fact_static fact.c -static -lqint -lq -lgmp -lpthread -lm -ldl
> 
> segfaults when I run it.

Yes, this is a bug. You could try to rebuild libqint after enabling the 
call to LTDL_SET_PRELOADED_SYMBOLS in libmain() (q.c, line 2211), but 
then you might have to link your qcwrapped program using libtool. If you 
try that please let me know how it works.

Albert

-- 
Dr. Albert Gr"af
Dept. of Music-Informatics, University of Mainz, Germany
Email:  Dr....@t-..., ag...@mu...
WWW:    http://www.musikwissenschaft.uni-mainz.de/~ag