lua-users-stdlib — Discussion of non-core standard libraries for Lua

Re: [Lua-users-stdlib] Upheaval (blame Jamie Webb!)

[Jamie W. <j...@jm...>]

On Wednesday 04 February 2004 10:58, Reuben Thomas wrote:
> > Seriously though, I realise it's not very general, but I have found it
> > useful.  I suppose it just depends on how large you want the library to
> > end up.
>
> That's a difficult question, of course: we have multiple goals. First, to
> reach into and improve the libraries actually shipped with Lua. Secondly,
> to add some core programming utils (this is mostly what we've discussed so
> far). Thirdly, to add lots of special purpose stuff (like Diego's mbox
> parser, which I was eulogising last night).

I'm wondering though if the special-purpose stuff wouldn't be better going 
straight into Cheia rather than stdlib.

>   1 Inventing a new "core" library and simply keeping it as small as
>     possible (perhaps setting a hard limit on number of lines of code,
>     which may sound silly, but could be helpful in focussing the mind on
>     what's important)

Perhaps some metrics on level of usage of each function would be more 
appropriate.

>   2 Putting everything strictly in table & string; this doesn't really
>     work, because a lot of stuff we have is really to do with functions,
>     not tables.

I agree.

>   3 Adding a few new libraries as necessary. For example, I propose to
>     abolish my set module, folding it back into table, but adding a vector
>     module is arguably good.

Mmm... except that most of what is in Lua table ought to be in vector.

> I'll end with a concrete proposal of what I think is best from the above:
> we should add "vector" and "function" to the current "string", "table"
> &c., and put all our additions into those four tables. One or two are
> tricky (are map and filter in function or table?) but they all fit in one
> of the four. Once that's done we can take another look.

Well, fn or something rather than function presumably. Given the argument 
order, the existing convention suggests that map and filter ought to be in 
function. Trouble with that is that then you can't have vector.map, list.map, 
table.map, etc. So, either the argument order needs changing (bad, because 
other languages have it that way round, and it makes the most sense for 
partial applicaton), the convention needs abandoning (might be bad, depending 
on this whole { __index = table } business), or map needs to be on vectors 
only (bad; not very forward-thinking).

 -- Jamie Webb

Re: [Lua-users-stdlib] Contribution

[Reuben T. <rr...@sc...>]

> Not especially, e.g.:
>
> list = { h = "Item 1", t = { h = "Item 2", t = { h = "Item 3", t = nil }}}
>
> Insertion at a random point in such a structure is O(1) if you've already
> found that point, whereas it's O(n) for vectors.

Right-o, I was being a bit dense.

> Trees also have their advantages at times (e.g. when keys are complex,
> since 'equal' Lua tables don't hash the same, or simply as a compromise
> between lookup and ordered insert speed). Just because we have this
> built-in hash-table doesn't mean that other classic data structures
> don't have their uses, and standard libraries for other langauges tend
> to implement them.

Although there's usually only any point in Lua for efficiency grounds.
I've never needed to implement anything else (although trees, as well as
lists, don't really need "implementing", you can just use keys 1 and 2 for
head and tail or left and right).

> Well yes, but I don't think you would want all tables being vectors by
> default, so arg would need to be treated separately.

I see...

> Yes, and also it should be possible to avoid building lists, concatenating
> them, then unpacking them, as in e.g. bind().

Are there primitives that could be used in Lua here? i.e. without having
to implement even a few core functions in Lua, but just by allowing Lua
more access to...I dunno, tuples?

> I think the Lua authors argument is that it is much harder to mess about with
> binary modules, so for simple stuff it's best if you don't have to, whereas
> it's very easy to download Lua code separately and dofile() it.

True but somewhat missing the point. There's a big difference in most
people's minds between stuff that's the standard distribution (gets under
the Not Invented Here barrier) and stuff that isn't.

> Python and Smalltalk have copy and deepcopy

That might be better than copy and clone? At least makes sense to some
people. But thinking genetically "clone" makes sense as deep copy.

-- 
http://www.mupsych.org/~rrt/ | violence, n.  bravery for cowards

Re: [Lua-users-stdlib] Contribution

[Jamie W. <j...@jm...>]

On Wednesday 04 February 2004 09:44, Reuben Thomas wrote:
> I meant Lua lists. Are you thinking of implementing a linked list type in
> C?

Not especially, e.g.:

list = { h = "Item 1", t = { h = "Item 2", t = { h = "Item 3", t = nil }}}

Insertion at a random point in such a structure is O(1) if you've already 
found that point, whereas it's O(n) for vectors. Trees also have their 
advantages at times (e.g. when keys are complex, since 'equal' Lua tables 
don't hash the same, or simply as a compromise between lookup and ordered 
insert speed). Just because we have this built-in hash-table doesn't mean 
that other classic data structures don't have their uses, and standard 
libraries for other langauges tend to implement them.

> Maybe this is something to sort out. If we get a coherent and compelling
> story, and implement it without dependencies on the rest of stdlib, it
> could get into the base distribution. The Lua authors themselves admitted
> (before 5.0) that the libraries hadn't had as much attention as they
> deserved, so the case for fine-tuning (which this essentially is) is still
> strong. LuaPlus addresses similar issues (e.g. metatables for all types).

Certainly worth a try.

> What about "for all tables"? Setting it for a particular bound variables
> seems a bit screwy. Also, there's a one-line fix to allow you to get at
> the metatable for tables (which already exists).

Well yes, but I don't think you would want all tables being vectors by 
default, so arg would need to be treated separately.

> You mean because of popping arguments off the Lua stack instead of
> stuffing them into a table? Point.

Yes, and also it should be possible to avoid building lists, concatenating 
them, then unpacking them, as in e.g. bind().

> Except that the point shouldn't just be to provide things you can't
> implement in Lua, but to provide things that make programming more
> productive. A lot of what we've done fits into that bracket: the very fact
> that we both line-for-line implemented many of the same functions suggests
> they have merit; plenty of programmers will not even know they exist.

I think the Lua authors argument is that it is much harder to mess about with 
binary modules, so for simple stuff it's best if you don't have to, whereas 
it's very easy to download Lua code separately and dofile() it.

> > > How about "copy" (for shallow) and "clone" (for deep)?
> >
> > Clone certainly sounds like a deep copy to me. Not so sure about copy
> > though. I think that could mean either, as evidenced by the terms deep
> > and shallow copy.
>
> I think I'll go with that for the present. I think that finding a better
> balance between short, obvious names and unambiguous ones will be hard,
> but of course I'm still open to suggestions.

Mmm... I really wish there was a consensus on this:

Java has clone for shallow or deep copy depending on the programmer's mood 
today.
C# has clone for deep copy
Python and Smalltalk have copy and deepcopy
ML typically has copy meaning deep copy
Ruby has dup for shallow copy and clone for something which doesn't appear to 
be either.
ADA uses 'all' for deep copy

So, I suppose copy and clone are as good as anything else. Everything's going 
to be confusing to someone.

 -- Jamie Webb

Whatever you're thinking, you're wrong.

Re: [Lua-users-stdlib] Upheaval (blame Jamie Webb!)

[Reuben T. <rr...@sc...>]

> Seriously though, I realise it's not very general, but I have found it
> useful.  I suppose it just depends on how large you want the library to
> end up.

That's a difficult question, of course: we have multiple goals. First, to
reach into and improve the libraries actually shipped with Lua. Secondly,
to add some core programming utils (this is mostly what we've discussed so
far). Thirdly, to add lots of special purpose stuff (like Diego's mbox
parser, which I was eulogising last night).

So far I've not really separated 1 and 2 (in a sense they shouldn't be,
because both are aiming at the goal of providing an improved standard
library that is actually shipped, or at least in our view should be
shipped, with the standard distribution).

However, it's easy to tell the difference between 1+2 and 3 (if you'll
forgive the pun): code for 1 & 2 have mostly been put in modules with the
same names as the existing C libraries, and the rest has gone in special
modules.

I've already removed some of my functions that I haven't used much or at
all; I think the first thing to do should be to try to find an optimum
"shape" for the core stuff. Options include

  1 Inventing a new "core" library and simply keeping it as small as
    possible (perhaps setting a hard limit on number of lines of code,
    which may sound silly, but could be helpful in focussing the mind on
    what's important)

  2 Putting everything strictly in table & string; this doesn't really
    work, because a lot of stuff we have is really to do with functions,
    not tables.

  3 Adding a few new libraries as necessary. For example, I propose to
    abolish my set module, folding it back into table, but adding a vector
    module is arguably good.

I'll end with a concrete proposal of what I think is best from the above:
we should add "vector" and "function" to the current "string", "table"
&c., and put all our additions into those four tables. One or two are
tricky (are map and filter in function or table?) but they all fit in one
of the four. Once that's done we can take another look.

> method() is to methods what bind() is to functions (bind_method() lies
> somewhere in the middle). I don't see how you can claim that one is useful
> but not the other. Personally, I use them both about equally, and from my
> libary use only map() more frequently.

Fair enough. Arguments like this are difficult to resolve until we've had
more experience with a larger community of programs and more lines of
code. There's scope for a little bloat in the interests of variety to
start with in the library as long as it's understood that it should get
smaller (in the core), not bigger. I don't think there's much of a problem
with outlying modules getting bigger.

Of course, this is always going to be the stuff that people argue over the
most, too.

-- 
http://www.mupsych.org/~rrt/ | golf, n.  a good walk spoiled (Twain)

Re: [Lua-users-stdlib] Contribution

[Reuben T. <rr...@sc...>]

> > But tables give you those operations and with lower complexity. What's the
> > problem?
>
> Linked lists have faster inserts and deletes,

I meant Lua lists. Are you thinking of implementing a linked list type in
C?

> It needs to be there though if you're going to do OOP, otherwise you might
> have a set containing the name of one of your methods.

I don't want to do OOP on sets any more.

> With the current Lua table stuff, one can set { __index = table } for a vector
> and have nice OO operations like t:getn(), and that's great, except that it's
> unreliable since any vectors a function receives need not have __index set.
> My problem is that it seems clear the the Lua authors intended the OO
> behaviour to be available for vectors, but failed to follow through. That
> makes me unsure as to what exactly the 'Lua way' is supposed to be.

Maybe this is something to sort out. If we get a coherent and compelling
story, and implement it without dependencies on the rest of stdlib, it
could get into the base distribution. The Lua authors themselves admitted
(before 5.0) that the libraries hadn't had as much attention as they
deserved, so the case for fine-tuning (which this essentially is) is still
strong. LuaPlus addresses similar issues (e.g. metatables for all types).

> What I'd like to see in the language I think is a) a way to set the default
> metatable for 'arg',

What about "for all tables"? Setting it for a particular bound variables
seems a bit screwy. Also, there's a one-line fix to allow you to get at
the metatable for tables (which already exists).

> Your map probably wouldn't be much different, but anything variadic can
> probably become noticeably faster in C.

You mean because of popping arguments off the Lua stack instead of
stuffing them into a table? Point.

> Most of the current Lua library functions are not implementable in Lua,
> or at least not efficiently, and that could be the criteria for any
> further additions.

Except that the point shouldn't just be to provide things you can't
implement in Lua, but to provide things that make programming more
productive. A lot of what we've done fits into that bracket: the very fact
that we both line-for-line implemented many of the same functions suggests
they have merit; plenty of programmers will not even know they exist.

> > How about "copy" (for shallow) and "clone" (for deep)?
>
> Clone certainly sounds like a deep copy to me. Not so sure about copy though.
> I think that could mean either, as evidenced by the terms deep and shallow
> copy.

I think I'll go with that for the present. I think that finding a better
balance between short, obvious names and unambiguous ones will be hard,
but of course I'm still open to suggestions.

-- 
http://www.mupsych.org/~rrt/ | impatience, n.  the urge to do nothing

Re: [Lua-users-stdlib] Contribution

[Jamie W. <j...@jm...>]

On Monday 02 February 2004 09:38, Reuben Thomas wrote:
> > I can imagine wanting a linked list, and then wanting the standard
> > operations on it.
>
> But tables give you those operations and with lower complexity. What's the
> problem?

Linked lists have faster inserts and deletes, and you can use tail() 
efficiently for nice recursive algorithms (I know you could just pass a start 
index around, but it's not as neat).

> > Everything's an untyped reference in Lua so all tables are monomorphic,
>
> Strings? Numbers? Threads? Functions? Booleans?

I meant that tables are monomorphic containers. All those types you list are 
still accessed using untyped references.

> I'd forgotten I had the set in a set member, and came up against that last
> night with some old code. This will be fixed.

It needs to be there though if you're going to do OOP, otherwise you might 
have a set containing the name of one of your methods.

> Depends what for. I want to leave the core tables stuff in the library as
> simple as possible so that everyone can use it in even their most basic
> scripts. That's certainly the only way the Lua authors are likely to get
> interested. The special-purpose stuff like getopt and parsers can quite
> happily be more OO: it's the sort of thing that attracts users because it
> already does most of their job, but only if they're doing that specific
> thing.

With the current Lua table stuff, one can set { __index = table } for a vector 
and have nice OO operations like t:getn(), and that's great, except that it's 
unreliable since any vectors a function receives need not have __index set. 
My problem is that it seems clear the the Lua authors intended the OO 
behaviour to be available for vectors, but failed to follow through. That 
makes me unsure as to what exactly the 'Lua way' is supposed to be.

What I'd like to see in the language I think is a) a way to set the default 
metatable for 'arg', b) distinguished __index and __method, and c) a way to 
type function parameters, so that we can write code something like this, but 
hopefully more efficient:

function foobar(x : Vector)
	-- Above line equivalent to writing 'x = Vector(x)' here
	-- can rely on x:map() working in here
end

Vector = setmetatable({
		new = function(self, ...)
			return setmetatable(arg, {
				__method = self,
				is_Vector = true })
			end
		map = function(v,f) -- etc.
	}, {
		__call = function(self, v)
			local m = getmetatable(v)
			if not m then
				-- This part optional
				-- Could just insist on 'proper' vectors
				assert(type(v) == "table", "...")
				-- Maybe do some checking to make sure that
				-- v looks like a Vector here.
				return setmetatable({}, {
					__index = v,
					__newindex = v,
					__method = self,
					is_Vector = true })
			end
			-- Any object can still behave like a vector by setting is_Vector
			assert(m.is_Vector, "...")
			return v
		end
	})

The distinguished __method has more general applications, and I think the 
generality of that typing scheme (the 'type' can be any functor) fits the Lua 
philosophy.

> I would really like to see proof that map being coded in C makes a
> difference: the amount of Lua code in it is tiny, and surely the vast
> majority of the time will be spent in the table construction even if it's
> not spent in the function.

Your map probably wouldn't be much different, but anything variadic can 
probably become noticeably faster in C.

> Peversely, it does seem almost more likely that the Lua team will look
> kindly on new C code than new Lua code...which is silly, because the
> maintenance cost of Lua code from version to version is so much lower.

Most of the current Lua library functions are not implementable in Lua, or at 
least not efficiently, and that could be the criteria for any further 
additions.

> How about "copy" (for shallow) and "clone" (for deep)?

Clone certainly sounds like a deep copy to me. Not so sure about copy though. 
I think that could mean either, as evidenced by the terms deep and shallow 
copy.

 -- Jamie Webb

[Lua-users-stdlib] Diego Nahab's mbox parser now in stdlib

[Reuben T. <rr...@sc...>]

Thanks to Diego's confirmation that his mbox code on the Wiki at
http://lua-users.org/wiki/MailBoxParsing is released under the MIT
license, I've added a Lua 5 version to stdlib. Now available in CVS at
http://cvs.sourceforge.net/viewcvs.py/lua-users/stdlib/modules/ (or at
least, will be as soon as it shows up; the really impatient can just check
out CVS).

I just tried it out on a 1700+ message mbox, and a few tests showed it
seeming to work flawlessly (it got the number of messages right, looking
at some headers seemed to show them in the right places &c.).

Thanks Diego!

-- 
http://www.mupsych.org/~rrt/ | Academics age by degrees

Re: [Lua-users-stdlib] Upheaval (blame Jamie Webb!)

[Jamie W. <j...@jm...>]

On Monday 02 February 2004 10:29, Reuben Thomas wrote:
> > Naming: yeah, dunno. Arity: I decided it made sense only for commutative
> > operators to be variadic.
>
> Whereas in e.g. APL they all are, and you get alternating differences with
> -. I'm not sure what's worse: providing variadic / (what is that for?) or
> not providing it (why doesn't my code work suddenly when I replace + with
> /?).

How about:

["/"] = function(a,b,c) assert(c == nil, "/ is binary only") return a/b end

At least you get a meaningful error that way.

> Seems a bit contrived. If you wanted those that had attribute foo < 4,
> you'd still need to make a function.

small = std.filter(std.compose(std.bind(std.op[">"], 4), std.getter("foo")), 
objects)

Seriously though, I realise it's not very general, but I have found it useful. 
I suppose it just depends on how large you want the library to end up.

> > Say I want to call method swizzle(5) on each of a list of (maybe
> > polymorphic) objects:
> >
> > results = std.map(std.method("swizzle", 5), objects)
>
> And again, that only works with constant values. If you wanted to call
> swizzle(self.foo+self.bar), again, you'd need a function.

method() is to methods what bind() is to functions (bind_method() lies 
somewhere in the middle). I don't see how you can claim that one is useful 
but not the other. Personally, I use them both about equally, and from my 
libary use only map() more frequently.

If I wanted self.foo+self.bar, then yes, I'd probably write a function (or 
make a swizzle_foo_bar() method). But in general where the values are not 
constant:

results = std.map(std.method("swizzle"), objects, params)

 -- Jamie Webb

Re: [Lua-users-stdlib] Upheaval (blame Jamie Webb!)

[Reuben T. <rr...@sc...>]

> Naming: yeah, dunno. Arity: I decided it made sense only for commutative
> operators to be variadic.

Whereas in e.g. APL they all are, and you get alternating differences with
-. I'm not sure what's worse: providing variadic / (what is that for?) or
not providing it (why doesn't my code work suddenly when I replace + with
/?).

> > The following I plain don't understand:
> >
> > 	function std.getter(name)      -- what's this for?
> > 	function std.method(name, ...) -- what's this for?
> > 	function std.bind_method(t, m) -- what's this for?
>
> Say I have a list of objects and I want to get a list of those which have
> attribute foo = true:
>
> footrue = std.filter(std.getter("foo"), objects)

Seems a bit contrived. If you wanted those that had attribute foo < 4,
you'd still need to make a function.

> Say I want to call method swizzle(5) on each of a list of (maybe polymorphic)
> objects:
>
> results = std.map(std.method("swizzle", 5), objects)

And again, that only works with constant values. If you wanted to call
swizzle(self.foo+self.bar), again, you'd need a function.

> > I'm dubious about the following:
> >
> > 	function std.map_tbl(f, t)     -- it is really wise to map over a table?
> > 	function std.filter_tbl(f, t)  -- these should only be used on
> > lists/arrays IMO
>
> Why not? Changing the keys is maybe not such a great idea, but there doesn't
> seem to be any reason not to map/filter over the values. Obviously the order
> is undefined, but that's just expected from any operation on tables. I don't
> really see why one shouldn't map (most likely injections) over sets either.

> Fair enough. I just didn't think of that; I've never come across reverseMap
> before. I don't see it in your library though.

No, I haven't needed it yet!

-- 
http://www.mupsych.org/~rrt/ | Academics age by degrees

Re: [Lua-users-stdlib] Contribution

[Reuben T. <rr...@sc...>]

> I can imagine wanting a linked list, and then wanting the standard operations
> on it.

But tables give you those operations and with lower complexity. What's the
problem?

> Everything's an untyped reference in Lua so all tables are monomorphic,

Strings? Numbers? Threads? Functions? Booleans?

> but maybe 'vector' would be a better name, following Java and C++.

Sounds good, as in "vector" = "resizable array". I'm certainly beginning
to agree that in Lua it's better to use your style of definitions rather
than explicit types ("table", "vector", "set").

> Not so nice to use ordinary tables: Set.add({ set = s }, e)
> That sort of not-really-encapsulation doesn't exactly set a shining example.

I'd forgotten I had the set in a set member, and came up against that last
night with some old code. This will be fixed.

> I think there are two options: go the current Lua way and don't encourage OO
> usage because it might not work, or in the longer term work with the Lua team
> to allow tables to be more effectively typed though a combination of
> convention and language support, so that OOP at this level becomes more
> feasible.

I think you can do both. Being more Lua-ish for basic types (like your
library) doesn't preclude being OO-ish for objects (which can contain and
be contained in the Lua-ish types quite happily). I've certainly written
code like this (even my library doesn't formalise lists, after all).

> I think the latter course would lead to a 'nicer' library,

Depends what for. I want to leave the core tables stuff in the library as
simple as possible so that everyone can use it in even their most basic
scripts. That's certainly the only way the Lua authors are likely to get
interested. The special-purpose stuff like getopt and parsers can quite
happily be more OO: it's the sort of thing that attracts users because it
already does most of their job, but only if they're doing that specific
thing.

> but obviously it depends on the Lua authors' take on all this.

My take so far is that they have no interest in it. I think the best shot
to get some really useful stuff in the standard distribution is with the
core functions like map, filter, join &c., introducing no new types.
Probably just want to put them all in table.

> Makes sense. It does seem a reasonable goal of the standard library to
> completely supplant the Lua libraries where their interfaces seem suboptimal,
> and then lobby for its inclusion (or inclusion of a subset) into Lua 6.
> Having this library in Lua would certainly improve its position w.r.t.
> setting conventions for these simple structures.

I agree completely.

> > Can't you just use trim in the latter case? If there are two many

Aargh. "two many"? I cringe with shame.

> In general, I agree. It seems like map() is pretty primitive though, and
> especially if this library were to replace the current Lua libraries it would
> need some amount of C code.

I would really like to see proof that map being coded in C makes a
difference: the amount of Lua code in it is tiny, and surely the vast
majority of the time will be spent in the table construction even if it's
not spent in the function.

Peversely, it does seem almost more likely that the Lua team will look
kindly on new C code than new Lua code...which is silly, because the
maintenance cost of Lua code from version to version is so much lower.

> > > shallow_copy: You've called it clone(). I used the more explicit name to
> > > distinguish it clearly from deep_copy().
> >
> > It's a pity, because I prefer the shorter name.
>
> There are probably shorter names for the two that would be sufficient to
> distinguish them. mimic() and duplicate() come to mind, but maybe some
> quality time spent with a thesaurus would yield something better.

How about "copy" (for shallow) and "clone" (for deep)?

Everything else: right on.

-- 
http://www.mupsych.org/~rrt/ | aphorism, n.  a wise lie

Re: [Lua-users-stdlib] Upheaval (blame Jamie Webb!)

[Jamie W. <j...@jm...>]

On Sunday 01 February 2004 19:03, Reuben Thomas wrote:
>         the functionalised operators +, -, .. &c. -- not sure how best to
>               -- include these, e.g. how to name them, how general they
>               -- really should be (arity &c.)

Naming: yeah, dunno. Arity: I decided it made sense only for commutative 
operators to be variadic.

> The following I plain don't understand:
>
> 	function std.getter(name)      -- what's this for?
> 	function std.method(name, ...) -- what's this for?
> 	function std.bind_method(t, m) -- what's this for?

Say I have a list of objects and I want to get a list of those which have 
attribute foo = true:

footrue = std.filter(std.getter("foo"), objects)

Say I want to call method swizzle(5) on each of a list of (maybe polymorphic) 
objects:

results = std.map(std.method("swizzle", 5), objects)

bind_method() as I said is just a convenience. It means I can write 
std.bind_method(object, "method", ...) rather than std.bind(object.method, 
object, ...). It just happens that I had some code where I needed to do that 
a lot.

> I'm dubious about the following:
>
> 	function std.map_tbl(f, t)     -- it is really wise to map over a table?
> 	function std.filter_tbl(f, t)  -- these should only be used on
> lists/arrays IMO

Why not? Changing the keys is maybe not such a great idea, but there doesn't 
seem to be any reason not to map/filter over the values. Obviously the order 
is undefined, but that's just expected from any operation on tables. I don't 
really see why one shouldn't map (most likely injections) over sets either.

> 	function std.conjunction(...)  -- this seems a bit specialised;
>                                           it's really reverseMap composed
>                                           with functionalised "and"

Fair enough. I just didn't think of that; I've never come across reverseMap 
before. I don't see it in your library though.

 -- Jamie Webb

Re: [Lua-users-stdlib] Contribution

[Jamie W. <j...@jm...>]

On Saturday 31 January 2004 20:35, Reuben Thomas wrote:
> And I consider arrays to be fixed length and monomorphic. Also, most of my
> list functions are indeed intended to be used on lists as lists (that
> functional style again). I'm not really sure which is better, but it's
> certainly better to be consistent.

I was thinking more of the complexity of operations. I think of lists as 
having O(n) random access, O(1) head and tail operations, etc. And while it 
seems unlikely that anyone would want anything more array-like than a table, 
I can imagine wanting a linked list, and then wanting the standard operations 
on it.

Everything's an untyped reference in Lua so all tables are monomorphic, but 
maybe 'vector' would be a better name, following Java and C++.

> Of course, with the Set code I give, you can use constructed sets directly
> if you want, and indeed pass ordinary tables to the methods.

Not so nice to use ordinary tables: Set.add({ set = s }, e)
That sort of not-really-encapsulation doesn't exactly set a shining example.

> > I decided against that approach because you can't rely in it, e.g. it
> > doesn't hold for 'arg', which is otherwise a list/array, and other
> > modules might not use the convention.
>
> ...and given that Lua doesn't have any real type system, maybe that's the
> best thing to do, rather than giving an illusion of it.

I think there are two options: go the current Lua way and don't encourage OO 
usage because it might not work, or in the longer term work with the Lua team 
to allow tables to be more effectively typed though a combination of 
convention and language support, so that OOP at this level becomes more 
feasible.
I think the latter course would lead to a 'nicer' library, but obviously it 
depends on the Lua authors' take on all this.

> table.concat is evilly named IMO. You either want string.join or
> fold(table, concat_function) when you use it.

Makes sense. It does seem a reasonable goal of the standard library to 
completely supplant the Lua libraries where their interfaces seem suboptimal, 
and then lobby for its inclusion (or inclusion of a subset) into Lua 6. 
Having this library in Lua would certainly improve its position w.r.t. 
setting conventions for these simple structures.

> > split_words: I distinguished this from split(), because usually when you
> > split on, e.g. commas, you want to preserve leading and trailing blank
> > fields, whereas usually when you split on whitespace, you don't.
>
> Can't you just use trim in the latter case? If there are two many
> functions half the users will never find the one they need and the others
> will just spend their time learning them all rather than getting work
> done.

Fair enough. I just had it because it's a common case for me.

> > map: My version is extended to allow multiple lists in parallel to be
> > mapped over (i.e. the transpose of your mapWith()).
>
> Or just zipWith, but with the lists as arguments rather than as a list of
> lists.

Ah yes.

> In general I'm against recoding in C, except for a very few primitives.
> It's error prone, and the average programmer can't inspect it.

In general, I agree. It seems like map() is pretty primitive though, and 
especially if this library were to replace the current Lua libraries it would 
need some amount of C code.

> > shallow_copy: You've called it clone(). I used the more explicit name to
> > distinguish it clearly from deep_copy().
>
> It's a pity, because I prefer the shorter name.

There are probably shorter names for the two that would be sufficient to 
distinguish them. mimic() and duplicate() come to mind, but maybe some 
quality time spent with a thesaurus would yield something better.

> Now I'm feeling rather daunted, however, mainly over all the
> table/list/set/array operations. I suggest the best way to proceed is to
> try to construct roughly the intersection of what we have between us (in
> terms of the size of the API), but with the union of functionality and,
> where possible, efficiency. How does that sound?

Great, if you think it's achievable.

> Partly I'm worried about alienating programmers without experience of
> functional languages by making the interface as aggressively functional as
> it is at the moment;

1) I've certainly heard it suggested that users with no existing programming 
experience can pick up imperative or functional styles with equal ease.

2) Professional programmers who haven't been exposed to a functional style 
really ought to be. It will help them write better code in any language.

3) I can't see much alternative. If you provide the directly useful 
higher-order functions (map, filter...), you naturally want the complete set 
of them (compose, bind...) because they complement each other. And the 
imperative alternative that first category is just to use control structures; 
no library required.

> Perhaps the terminology might need to be refined: map is a rather
> mathematical word.

OTOH every other language calls it map. It seems to me that new users are 
going to have to learn a name for the function whatever you call it, whereas 
programmers from other languages will be confused/annoyed if it isn't called 
what they expect.

 -- Jamie Webb

[Lua-users-stdlib] Upheaval (blame Jamie Webb!)

[Reuben T. <rr...@sc...>]

I have just committed some changes provoked by Jamie Webb's submission of
his std.lua library, which was mostly aimed at core functionality
(manipulation of arrays, sets, tables and functions).

So far I have incorporated all the stuff which matched existing functions.
I've also removed some functions (from my stuff) that didn't look that
useful and made other miscellaneous fixes and improvements that occurred
to me as I read through the code.

I'm left mostly with the functions that don't match anything I have, which
I'll divide into groups: those I have no problems with and will go ahead
and add later, those I don't think are necessary, and those I don't know
what to do with:

The following are fine:

	function std.succ(x)
	function std.generate(f, start, length)
        function std.deep_copy(t)   (but what to call it and shallow_copy?)
	function std.intersect(a, b)    -- I'll reimplement my set module
	function std.difference(a, b)   -- with these in mind, probably
                                        -- getting rid of my set type

The following look good but I'm not sure how best to handle them:

	function std.dump(value, key, fh, depth, pr, done) -- better done
              -- by reimplementing my tostring using a visitor and then
              -- reusing the visitor for dump
        the functionalised operators +, -, .. &c. -- not sure how best to
              -- include these, e.g. how to name them, how general they
              -- really should be (arity &c.)

The following I plain don't understand:

	function std.getter(name)      -- what's this for?
	function std.method(name, ...) -- what's this for?
	function std.bind_method(t, m) -- what's this for?

I'm dubious about the following:

	function std.map_tbl(f, t)     -- it is really wise to map over a table?
	function std.filter_tbl(f, t)  -- these should only be used on lists/arrays IMO

	function std.conjunction(...)  -- this seems a bit specialised;
                                          it's really reverseMap composed
                                          with functionalised "and"

-- 
http://www.mupsych.org/~rrt/ | perfect, a.  unsatirizable

Re: [Lua-users-stdlib] Contribution

[Reuben T. <rr...@sc...>]

By the way, for cons, what's wrong with:

function cons (x, l)
  return {x, unpack (l)}
end

(and similarly for appending one item)

Re: [Lua-users-stdlib] Contribution

[Reuben T. <rr...@sc...>]

> Firstly, functions in my library do not modify their arguments unless you use
> the *_inplace versions which exist for some functions. I think this avoids
> confusion and having both versions could reduce unnecessary copy operations.

I pretty much don't modify arguments unless the function is obviously
intended to do that. I've never had performance problems, but I haven't
run big or long-running programs. I've tried to encourage a functional
style in general.

> Also, I use the notation 'array' to refer to the tables you call 'lists'
> because it seems to me that they have access semantics more like arrays,
> whereas lists I consider to mean sequential structures such as
> functional-style recursive lists (or more generally linked lists).

And I consider arrays to be fixed length and monomorphic. Also, most of my
list functions are indeed intended to be used on lists as lists (that
functional style again). I'm not really sure which is better, but it's
certainly better to be consistent.

> I didn't see the need for a Set object.

Mainly metamethods.

Another tension I have sensed in building the libraries this far is
between using Lua's freedom to do anything with tables for easy dynamic
polymorphism on the one hand, against the propensity to get hard-to-find
bugs this way, and the difficulty of actually knowing you have a function
that does what you want (e.g. transpose = zip = unzip). Also type
"safety". Also object orientation.

Of course, with the Set code I give, you can use constructed sets directly
if you want, and indeed pass ordinary tables to the methods.

> I decided against that approach because you can't rely in it, e.g. it
> doesn't hold for 'arg', which is otherwise a list/array, and other
> modules might not use the convention.

...and given that Lua doesn't have any real type system, maybe that's the
best thing to do, rather than giving an illusion of it.

> bind: I think this name better describes what the function does.

Agreed.

> join: Does what you've called list.concat(). I chose join() to avoid confusion
> with table.concat(), since that also acts on lists.

table.concat is evilly named IMO. You either want string.join or
fold(table, concat_function) when you use it.

> split_words: I distinguished this from split(), because usually when you split
> on, e.g. commas, you want to preserve leading and trailing blank fields,
> whereas usually when you split on whitespace, you don't.

Can't you just use trim in the latter case? If there are two many
functions half the users will never find the one they need and the others
will just spend their time learning them all rather than getting work
done.

> map: My version is extended to allow multiple lists in parallel to be mapped
> over (i.e. the transpose of your mapWith()).

Or just zipWith, but with the lists as arguments rather than as a list of
lists.

> There's a performance hit though for the typical case. That could be
> virtually eliminated by coding in C (better vararg handling), and this a
> very commonly useful function, so I think that would be the best option
> in the long run. In the meantime, maybe this should be added but under a
> different name.

In general I'm against recoding in C, except for a very few primitives.
It's error prone, and the average programmer can't inspect it.

> tail: You call this behead().

I was thinking of in-place modifications for argument list processing when
I wrote it (originally I called it shift). I tend to agree with you about
the rest.

> My version is missing the extra numerical argument though.

Easily fixed.

> shallow_copy: You've called it clone(). I used the more explicit name to
> distinguish it clearly from deep_copy().

It's a pity, because I prefer the shorter name.

> deep_copy: I don't think you have this.

No, I've never needed it.

> size: Returns the size of a table or set (not list/array). I'm surpised I
> didn't see this one in your libary. Have I missed something?

I think I've never needed anything more than table.empty.

> op[*]: Function forms of the operators, extended to many arguments where
> appropriate. I used the symbols rather than naming them to avoid confusion
> (subtract or minus?). Not entirely sure if that was worth it though. Useful
> in parameters to map, filter, generate...

I like that.

> dump: Dumps the contents of a table for debugging purposes.

I'd like to have a pretty printer on my extended tostring for that.

Anything I haven't commented on I agree with.

Now I'm feeling rather daunted, however, mainly over all the
table/list/set/array operations. I suggest the best way to proceed is to
try to construct roughly the intersection of what we have between us (in
terms of the size of the API), but with the union of functionality and,
where possible, efficiency. How does that sound?

Partly I'm worried about alienating programmers without experience of
functional languages by making the interface as aggressively functional as
it is at the moment; partly I feel this is a great opportunity to educate,
and I am also encouraged by reading the book "A Small Matter of
Programming" about "programming" by users (e.g. in spreadsheets (formulae,
not macros) and CAD systems (templates)) which mentions that the greatest
sticking point that most users have when trying to learn anything like a
general-purpose programming language (e.g. spreadsheet macro language) is
with the control-flow stuff. The great thing about map, zip & friends is
that you avoid control-flow (other than if-then-else, which was found
easier to grasp). Encouraging Lua programmers to give their users
functional primitives that they can use to write typically one-to-three
line programs without needing to understand while or for might be a big
win, albeit a rather visionary one.

Perhaps the terminology might need to be refined: map is a rather
mathematical word. But the idea that if you have a list of things and a
function that works on one then you can say

    map (f, list)

is much more obvious than trying to explain

    for x in list
      f (x)
    end

-- 
http://www.mupsych.org/~rrt/ | certain, a.  insufficiently analysed

Re: [Lua-users-stdlib] Contribution

[Jamie W. <j...@jm...>]

On Friday 30 January 2004 20:22, Reuben Thomas wrote:
> > Attached is the standard library I've been gathering. It's rather smaller
> > than yours and duplicates some of the contents, but it also has some new
> > functions which you might want to consider for inclusion.
>
> Thanks very much for this. It does look as though there are some handy
> additions to stdlib here.
>
> Hi. I've had a look, and there's a lot of overlap, but also differences of
> approach. It would be very helpful if you could tell me which functions
> are new (i.e. not in stdlib) and which you think are better (either in
> functionality, documentation, or implementation) in your version, as I'm
> bound to miss things otherwise.

Ok, here goes:

Firstly, functions in my library do not modify their arguments unless you use 
the *_inplace versions which exist for some functions. I think this avoids 
confusion and having both versions could reduce unnecessary copy operations.

Also, I use the notation 'array' to refer to the tables you call 'lists' 
because it seems to me that they have access semantics more like arrays, 
whereas lists I consider to mean sequential structures such as 
functional-style recursive lists (or more generally linked lists).

I didn't see the need for a Set object. I just defined a set as a table with 
all values true. Given Lua lists/arrays, this seems more in keeping. I see 
you've made it an object because you want to, e.g. set 'table' as the __index 
table for tables, and that doesn't work for sets. I decided against that 
approach because you can't rely in it, e.g. it doesn't hold for 'arg', which 
is otherwise a list/array, and other modules might not use the convention.

succ: mainly convenient as an argument to generate()

curry: The function that you call curry() does what I have called bind(), i.e. 
partially applies a function. My curry() function converts the function into 
a curried version. Certainly much less useful than bind(), but notable as 
something distinct. Probably needs rewriting in C eventually for speed.

bind: I think this name better describes what the function does.

bind_method: Just a convenience to avoid passing the table name twice when 
binding a table method.

conjunction: Mainly useful to build parameters to filter()

join: Does what you've called list.concat(). I chose join() to avoid confusion 
with table.concat(), since that also acts on lists.

split: I think my implementation is probably more efficient (although 
admittedly I haven't tested it very thoroughly).

split_words: I distinguished this from split(), because usually when you split 
on, e.g. commas, you want to preserve leading and trailing blank fields, 
whereas usually when you split on whitespace, you don't.

ltrim, rtrim, trim: Remove edge whitespace. I don't think these are present in 
your library.

map: My version is extended to allow multiple lists in parallel to be mapped 
over (i.e. the transpose of your mapWith()). There's a performance hit though 
for the typical case. That could be virtually eliminated by coding in C 
(better vararg handling), and this a very commonly useful function, so I 
think that would be the best option in the long run. In the meantime, maybe 
this should be added but under a different name.

map_tbl, filter_tbl: Like map and filter, but for general tables.

tail: You call this behead(). My version returns a new table (and there is no 
tail_inplace() version since the difference in efficiency is minimal and I 
was too lazy). tail() is the more common name for this function, esp. in 
functional programming circles. My version is missing the extra numerical 
argument though.

cons: Adds a single value to the start of an array. Just a common special case 
of join(). Again, it's a functional thing.

shallow_copy: You've called it clone(). I used the more explicit name to 
distinguish it clearly from deep_copy().

deep_copy: I don't think you have this. I've just noticed some bugs in my 
implementation though. Corrected and better tested implementation below.

getter: Useful to build arguments to map() and filter(), e.g. 
std.map(std.getter("x"), t) returns { t[1].x, t[2].x, ... }.

method: Useful to build arguments to map() and filter(), e.g. 
std.map(std.method("x", y), t) returns { t[1]:x(y), t[2]:x(y), ... }. I use 
this one all the time.

generate: Builds sequences, i.e. generate(f, s, n) returns the n-element table 
{ s, f(s), f(f(s)), f(f(f(s))), ... }.

override: Serves as your Set.union() and table.merge().

intersect, difference: Again, work on sets and tables.

size: Returns the size of a table or set (not list/array). I'm surpised I 
didn't see this one in your libary. Have I missed something?

op[*]: Function forms of the operators, extended to many arguments where 
appropriate. I used the symbols rather than naming them to avoid confusion 
(subtract or minus?). Not entirely sure if that was worth it though. Useful 
in parameters to map, filter, generate...

dump: Dumps the contents of a table for debugging purposes.

 -- Jamie Webb

	function std.deep_copy(t)
		local r = {}
		local d = { [t] = r }
		local function h(o,x)
			for k,v in pairs(x) do
				if type(v) == "table" then
					if not d[v] then
						d[v] = {}
						local q = h(d[v], v)
						o[k] = q
					else
						o[k] = d[v]
					end
				else
					o[k] = v
				end
			end
			return o
		end
		return h(r, t)
	end

Re: [Lua-users-stdlib] bug in pickle, suggestions

[Reuben T. <rr...@sc...>]

> function pickle (x)
> ...
>      return format ("%q", x)
> --
>
> That should be "string.format", I believe.

Indeed. Another one I haven't tested since moving to Lua 5.

> function round(x, n_digits)
>    local p_10 = 10 ^ n_digits
>    return math.floor(x * p_10 + 0.5) / p_10
> end

Great.

> as well as second the trim, rtrim, ltrim, and join functions for
> inclusion eventually.

Noted.

> I've written implementations for most of the Common Lisp sequence
> functions, using a final table argument to pass the keyword arguments,
> if anyone is interested.

I suspect that would be good, but to be really good it would be integrated
with the existing list and table functions.

In general I don't want to just amass a whole pile of code, but a small
pile with rich functionality.

-- 
http://www.mupsych.org/~rrt/ | certain, a.  insufficiently analysed

[Lua-users-stdlib] ldoc -v

[Kaos <ka...@ex...>]

Hi,

When I run 'lua ldoc -v' i get the version string as expected, but it 
never quits (unless I press ctrl-c, or ctrl-z enter).
Expected behaviour?

//Andreas

[Lua-users-stdlib] bug in pickle, suggestions

[Johann H. <jhi...@ya...>]

function pickle (x)
...
     return format ("%q", x)
--

That should be "string.format", I believe.

I'd suggest:

function round(x, n_digits)
   local p_10 = 10 ^ n_digits
   return math.floor(x * p_10 + 0.5) / p_10
end

as well as second the trim, rtrim, ltrim, and join functions for 
inclusion eventually.

It's been a while since I've used lua heavily, so we'll just have to 
wait and see what I come up with.

I've written implementations for most of the Common Lisp sequence 
functions, using a final table argument to pass the keyword arguments, 
if anyone is interested.

-Johann

Re: [Lua-users-stdlib] Contribution

[Reuben T. <rr...@sc...>]

> Attached is the standard library I've been gathering. It's rather smaller than
> yours and duplicates some of the contents, but it also has some new functions
> which you might want to consider for inclusion.

Thanks very much for this. It does look as though there are some handy
additions to stdlib here.

Hi. I've had a look, and there's a lot of overlap, but also differences of
approach. It would be very helpful if you could tell me which functions
are new (i.e. not in stdlib) and which you think are better (either in
functionality, documentation, or implementation) in your version, as I'm
bound to miss things otherwise.

-- 
http://www.mupsych.org/~rrt/ | Eschatology Generally Breeds Dire Fanaticism

[Lua-users-stdlib] Contribution

[Jamie W. <j...@jm...>]

Reuben,

Attached is the standard library I've been gathering. It's rather smaller than 
yours and duplicates some of the contents, but it also has some new functions 
which you might want to consider for inclusion.

 -- Jamie Webb