state-threads-devel — State Threads developers' list

Re: [ST-devel] Re: packaging libst: autoconf and libtool

[Claus A. <sta...@es...>]

On Thu, Dec 27, 2001, Gene wrote:

> I generally agree with you -- autoconf/libtool are good for large
> application projects.  But for a small development library, I think,
> they are overkill.

What about the software that uses state-threads?  I would like to
use it (even though I haven't figured out the license issues) and
I would prefer if the entire system just "automagically" builds on
many OS. Hence I appreciate the efforts to use autoconf. However,
I agree that there should be no performance impact, i.e., runtime
decisions what to use. Maybe some "use these switches on this OS"
cases can be integrated? Currently that's done by hand by selecting
the correct OS target in the Makefile.

[ST-devel] Re: packaging libst: autoconf and libtool

[Gene <ge...@pa...>]

"Wesley W. Terpstra" wrote:
> 
> 
> I would argue that autoconf/libtool are good things (tm) for keeping the
> learning curve simple, getting correct build stuff for free, and simpler
> integration.
>

I generally agree with you -- autoconf/libtool are good for large
application projects.  But for a small development library, I think,
they are overkill.

There was a discussion on slashdot last month about good and evil
of autoconf/libtool:

http://slashdot.org/article.pl?sid=01/11/27/2259253&mode=thread

People brought up many interesting points there.

An autoconf/automake/libtool rant can be found at:

http://www.ohse.de/uwe/articles/aal.html

Re: [ST-devel] Re: packaging libst: autoconf and libtool

[Wesley W. T. <we...@te...>]

On Wed, Dec 26, 2001 at 01:48:00PM -0800, Gene wrote:
(reordered to make my reply coherent)
>
> I don't want to get into philosophical discussion about
> good and evil of the autoconf and libtool, but here is my
> $0.02 why they are a wrong way to go in the ST case.
>
> <snip arguements about detection vs. the sure thing>
>
> I believe the right thing to do for automated builds is to
> figure out OS and then just invoke make with the appropriate
> default arguments.  For example, if OS is Solaris 2.x, just
> call gmake solaris-optimized
>
> Minimum guesswork.  The only step which is currently missing
> is figuring out the OS.

I began to agree with this point of view when I was more than half-done the
conversion. However, to see the final result, I kept on chugging. Now,
however, I am inclined to agree with you about the truly system dependent
stuff: md.h is nice and simple

Indeed, the reason I finally came to agree with this point of view is the
MD_INIT_CONTEXT function: it really is different on each system and you have
to know what it should be. In fact, this is what the last step of my
configure script does right now at the bottom of acinclude.m4 to determine
how to context switch.

Indeed, my early worries about cross-compiling breaking are addressed by
this scheme. You know the target platform, so you don't need to guess.

On the other hand, for those things which you can reliably detect without
knowing which platform you're on, I think the effort should be made.
It's easier to maintain then having to go look at each platform.
Further, autoconf provides a means for getting the target platform.

> If the -fpic flag has to be added on Linux, it should be added in the
> appropriate Makefile section.

Here I disagree; I think automake/libtool/autoconf should be used:

1. They provide a set of options that are familiar to more users
2. They ease adoption and integration with other projects because most
developers are familiar with how to deal with these proejcts over
hand-rolled build systems
3. They are actively maintained by people who know more than us about:
a: building libraries properly on various platforms
b: what things users need to be able to control
c: what things will work with various linkers/compiler/os combinations

To summarize, I agree that the context switching mechanism should not be
guessed at. I intend to change this as soon as I fly home. I think those
things that can be guessed reliably/determined efficiently at runtime should
be.

I would draw the division as follows:
	should be looked up based on machine type (md.h):
		MD_SETJMP
		MD_LONGJMP
		MD_INIT_CONTEXT
		MD_ALWAYS_UNSERIALIZED_ACCEPT
	should be determined at runtime (at no cost):
		MD_STACK_GROWS_UP/DOWN
		MD_ACCEPT_NB_INHERITED
	should be determined without use of architecture:
		MD_UTIME
		MD_USE_SYSV/BSD_ANON_MMAP
		MD_EPOLL/KQUEUE/etc :-)

I would argue that autoconf/libtool are good things (tm) for keeping the
learning curve simple, getting correct build stuff for free, and simpler
integration.

That's my $0.02 CA. :-)

-- 
Wesley W. Terpstra <we...@te...>

Re: [ST-devel] Re: autoconf and libst

[Lev W. <vl...@ne...>]

Gene wrote:

> Lev Walkin wrote:
> 
> I never said that.  I 100% agree with you that modern fast interface
> is better than the older slower one.  I don't think, however, that
> application's performance gains will be "huge".


Even 20% conservative gain will be better to achieve, than save
0.01% of performance on avoided system call.


-- 
Lev Walkin
vl...@ne...

Re: [ST-devel] Re: autoconf and libst

[Gene <ge...@pa...>]

Lev Walkin wrote:
> But why do YOU think that going
> with an old good poll instead of modern event-oriented interfaces
> is benefit?
> 

I never said that.  I 100% agree with you that modern fast interface
is better than the older slower one.  I don't think, however, that
application's performance gains will be "huge".

--Gene

Re: [ST-devel] autoconf and libst

[Lev W. <vl...@ne...>]

Mike Abbott wrote:

> I am all for adding kqueue and/or /dev/poll support to st.  I can't do
> it (swamped).  Are you volunteering?


Probably, yes. If it will not be done in a nearest future, I'll probably
do that. No promises.


>>There are a big area to improve, besides minimizing the number
>>of system calls.
>>
> 
> Besides {select,poll,kqueue,/dev/poll}, what is a big area of st needing
> improvement?


-- 
Lev Walkin
vl...@ne...

Re: [ST-devel] Re: autoconf and libst

[Lev W. <vl...@ne...>]

Gene wrote:

> Lev Walkin wrote:
> 
>>>>poll() is faster than select() - is this wrong?
>>>>
>>>>
>>>In addition to the other analysis offered of relative speed, select has
>>>the feature that multiple threads waiting for the same fd don't change
>>>the system call.  With poll, each waiting thread has its own pollfd even
>>>if all of the fds are the same.  So the right choice (poll vs. select)
>>>depends on not only the total number of fds but also their expected use.
>>>
>>Again: if we want to make st even faster (20% to TIMES faster), we should
>>
>>consider build an independent scheme, and implement a modern techniques,
>>like kqueue() or /dev/poll. We will give the ST the speed and the
>>scalability of state-driven kqueue() (or /dev/poll)'based applications.
>>
>>It will be a really huge improvement, not even comparable with omission
>>of a system call one time per thread life.
>>
>>
> 
> Oh no, not this poll vs. /dev/poll vs. kqueue discussion again!!! :)
> 
> Why do you think that optimizing the idle state would bring you a
> "huge improvement"?


Because I've read a number of papers related the impact of this
technologies to a _common_ types of services: http daemons, et
cetera. Everybody involved in this seems to agree that converting
a network events (NIC interrupts) into a states, than converting
these states back into events (poll()/select() then do the
event-like processing) is not a very good idea with the number
of sockets greater than ONE. But why do YOU think that going
with an old good poll instead of modern event-oriented interfaces
is benefit?

-- 
Lev Walkin
vl...@ne...

Re: [ST-devel] autoconf and libst

[Mike A. <mj...@at...>]

I am all for adding kqueue and/or /dev/poll support to st.  I can't do
it (swamped).  Are you volunteering?

> There are a big area to improve, besides minimizing the number
> of system calls.

Besides {select,poll,kqueue,/dev/poll}, what is a big area of st needing
improvement?

[ST-devel] Re: autoconf and libst

[Gene <ge...@pa...>]

Lev Walkin wrote:
> >>poll() is faster than select() - is this wrong?
> >>
> >
> > In addition to the other analysis offered of relative speed, select has
> > the feature that multiple threads waiting for the same fd don't change
> > the system call.  With poll, each waiting thread has its own pollfd even
> > if all of the fds are the same.  So the right choice (poll vs. select)
> > depends on not only the total number of fds but also their expected use.
> 
> Again: if we want to make st even faster (20% to TIMES faster), we should
> 
> consider build an independent scheme, and implement a modern techniques,
> like kqueue() or /dev/poll. We will give the ST the speed and the
> scalability of state-driven kqueue() (or /dev/poll)'based applications.
> 
> It will be a really huge improvement, not even comparable with omission
> of a system call one time per thread life.
> 

Oh no, not this poll vs. /dev/poll vs. kqueue discussion again!!! :)

Why do you think that optimizing the idle state would bring you a
"huge improvement"?

[ST-devel] Re: packaging libst: autoconf and libtool

[Gene <ge...@pa...>]

Hi all,

I don't want to get into philosophical discussion about
good and evil of the autoconf and libtool, but here is my
$0.02 why they are a wrong way to go in the ST case.

Trying to replace things that are known to work best on a
given platform with a guess just to fit into some artificial
build framework is a WRONG thing to do.  This would be a
serious step backward.

Guess is only good when you have nothing else and you bear
no responsibility for the result.  ST has its roots in the
industrial software and shares its philosophy -- you gotta
be sure that your software works properly on all supported
platforms before you ship it to customer.

ST already has all the (hopefully) best settings for each
supported platform.  Why would you want to revert it to
a guess?  If the -fpic flag has to be added on Linux, it
should be added in the appropriate Makefile section.

Some fundamental things (like thread context initialization)
can never be guessed 100% correctly in all cases.

I believe the right thing to do for automated builds is to
figure out OS and then just invoke make with the appropriate
default arguments.  For example, if OS is Solaris 2.x, just
call

gmake solaris-optimized

Minimum guesswork.  The only step which is currently missing
is figuring out the OS.

Thanks,
Gene

Re: [ST-devel] autoconf and libst

[Lev W. <vl...@ne...>]

Mike Abbott wrote:

> [I missed the original post on this subject due to @Home cratering and
> the list archive seems not to work, so sorry for the late reply.]

[skip]

> 	3.3 Performance
> 
> 	Performance is one of the library's main objectives.
> 	The State Threads library is implemented to minimize the
> 	number of system calls and to make thread creation and
> 	context switching as fast as possible. ...
> 
> 	(http://state-threads.sourceforge.net/docs/st.html)


Yes, yes, yes. But the general overhead of using a system call is
only TWO-THREE times worse of using a malloc(), for example.
There are a big area to improve, besides minimizing the number
of system calls.

> On the other hand I welcome packaging efforts.  Casual users frown on
> having to compile software so providing rpms, srpms, and other popular
> package formats will expand the project's audience.
> 
> Now to answer some specific questions:
> 
> 
>>MD_ACCEPT_NB_INHERITED:
>>Is it really such a performance hit?
>>
> 
> Yes.
> 
> 
>>MD_GET_UTIME:
>>	Use them in this order:
>>	gettimeofday
>>	read_real_time & time_base_to_time
>>	syssgi & clock_gettime
>>	gethrtime
>>	ftime
>>	fail with an error
>>
> 
> If you intend to probe for these functions in this order and to use the
> first match your performance will suffer.  clock_gettime on SGI systems
> is a memory load not a system call and is therefore much faster than
> gettimeofday.  I assume gethrtime is similar.  gettimeofday should be
> the last resort.
> 
> 
>>You should consider using internal
>>__OS_Type__ variables in header or code files to protect this place.
>>
> 
> Ugh, I strongly prefer the current scheme in md.h.  All the
> system-dependent definitions are in one place rather than scattered
> about.


It was for case of using autoconf.

>>Who is the official upstream maintainer?
>>
> 
> Please define this term.  Gene and I are the ultimate owners of the
> project.  I don't think sourceforge provides a "state-threads-owner"
> mail alias so snarf our sourceforge addresses from the project summary
> page.
> 
> 
>>poll() is faster than select() - is this wrong?
>>
> 
> In addition to the other analysis offered of relative speed, select has
> the feature that multiple threads waiting for the same fd don't change
> the system call.  With poll, each waiting thread has its own pollfd even
> if all of the fds are the same.  So the right choice (poll vs. select)
> depends on not only the total number of fds but also their expected use.


Again: if we want to make st even faster (20% to TIMES faster), we should

consider build an independent scheme, and implement a modern techniques,
like kqueue() or /dev/poll. We will give the ST the speed and the
scalability of state-driven kqueue() (or /dev/poll)'based applications.

It will be a really huge improvement, not even comparable with omission
of a system call one time per thread life.


-- 
Lev Walkin
vl...@ne...

Re: [ST-devel] autoconf and libst

[Mike A. <mj...@at...>]

[I missed the original post on this subject due to @Home cratering and
the list archive seems not to work, so sorry for the late reply.]

I applaud attempts to simplify and proliferate state threads but I have
doubts about the need for throwing autoconf into the mix.  For a large
and complex project autoconf has merit but for a tiny little library
like state threads it's overkill.  The machine-dependent definitions in
md.h take only a couple dozen lines each and the set of variables that
needs to be researched and defined to add new systems is small enough
that doing it manually is not arduous.  I'm not sure but isn't autoconf
itself larger than the whole st library?

Recent mail on this list about the auto-building subject ask questions
like "is adding a single system call to thread creation really a bad
thing?" and the like.  The answer is an unqualified yes, it is horrible.
The introductory paper says:

	3.3 Performance

	Performance is one of the library's main objectives.
	The State Threads library is implemented to minimize the
	number of system calls and to make thread creation and
	context switching as fast as possible. ...

	(http://state-threads.sourceforge.net/docs/st.html)

On the other hand I welcome packaging efforts.  Casual users frown on
having to compile software so providing rpms, srpms, and other popular
package formats will expand the project's audience.

Now to answer some specific questions:

> MD_ACCEPT_NB_INHERITED:
> Is it really such a performance hit?

Yes.

> MD_GET_UTIME:
> 	Use them in this order:
> 	gettimeofday
> 	read_real_time & time_base_to_time
> 	syssgi & clock_gettime
> 	gethrtime
> 	ftime
> 	fail with an error

If you intend to probe for these functions in this order and to use the
first match your performance will suffer.  clock_gettime on SGI systems
is a memory load not a system call and is therefore much faster than
gettimeofday.  I assume gethrtime is similar.  gettimeofday should be
the last resort.

> You should consider using internal
> __OS_Type__ variables in header or code files to protect this place.

Ugh, I strongly prefer the current scheme in md.h.  All the
system-dependent definitions are in one place rather than scattered
about.

> Who is the official upstream maintainer?

Please define this term.  Gene and I are the ultimate owners of the
project.  I don't think sourceforge provides a "state-threads-owner"
mail alias so snarf our sourceforge addresses from the project summary
page.

> poll() is faster than select() - is this wrong?

In addition to the other analysis offered of relative speed, select has
the feature that multiple threads waiting for the same fd don't change
the system call.  With poll, each waiting thread has its own pollfd even
if all of the fds are the same.  So the right choice (poll vs. select)
depends on not only the total number of fds but also their expected use.

Re: [ST-devel] autoconf and libst

[Lev W. <vl...@ne...>]

Wesley W. Terpstra wrote:

> On Tue, Dec 25, 2001 at 02:50:46PM -0800, Lev Walkin wrote:
> 
>>>MD_USE_SYSV/BSD_ANON_MMAP:
>>>	Idea 1:
>>>#ifdef MAP_ANON
>>>do BSD stuff
>>>#els
>>>do SYSV stuff
>>>#endif
>>>
>>>	Idea 2:
>>>always open /dev/zero
>>>#ifdef MAP_ANON
>>>flags |= MAP_ANON;
>>>#endif
>>>mmap
>>>
>>It is a bad idea to invoke useless open() call with mmap(), if we
>>can avoid it.
>>
> 
> Ok, that doesn't rule out idea 1 though. Does MAP_ANON being defined
> guarantee it is available and works?


I think, yes.

> Alternately, if build system == host system: run a check for sysv/bsd
> semantics. If cross compiling fall back to idea 2 or malloc. (whichever is
> better)


make a ./configure --able parameter.

> Again, what's wrong with malloc? I thought it uses mmap for large
> allocations internally?


In GNU libc - yes, it does. In other systems - generally NO.

>>>MD_ACCEPT_NB_INHERITED:
>>>
>>>	Why not just assume it's not inherited and always reset the thing
>>>nonblocking? Is it really such a performance hit?
>>>
>>System call is always a performance hit. But related with other
>>types of activities during thread initialization, it is almost nothing.
>>
> 
> Ok,
> 
> io.c:
> static int st_nb_inherited = -1;
> 
> void st_determine_if_nb_inherit(int fd)
> {
> 	long flags = fcntl(fd, F_GETFL);
> 	if ((flags & O_NONBLOCK) != 0)
> 		st_nb_inherited = 1;
> 	else
> 		st_nb_inherited = 0;
> }
> 
> ...
> // code using #ifdef now:
> if (st_nb_inherited == -1) st_derermine_if_nb_inherit(sock);
> if (st_nb_inherited) _st_newfd_new(sock, 0, 1);
> else                 _st_newfd_new(sock, 1, 1);
> ...
> EOF
> 
> This should address the performance issue...?


How ugly... But OK.


>>>MD_ALWAYS_UNSERIALIZED_ACCEPT:
>>>
>>>	I don't know enough about this to write a test. Could someone give
>>>me a hand?
>>>
>>Actually, it is somewhat hard to write a multiprocess test of such
>>a nature within automake framework. You should consider using internal
>>__OS_Type__ variables in header or code files to protect this place.
>>
> 
> Hrm. Ok then, which OSs need serialized accept?


Original md.h has the description of it.

> One last thing: I have it using poll() whenever the function is available
> since I am under the belief that poll() is faster than select() - is this
> wrong?

Well. This is a good question. Generally, neither is faster.
If you have 10 active file descriptors to handle, poll() will be better,
because it transfers only 80 bytes to kernel. If you have 50 active
descriptors, select() will do better because it will transfer to kernel
only 384 bytes (instead of poll()'s 400). But you will be not able to
supply more than 1024 descriptors to select() without recompiling
kernel on most systems.
Another bad thing is that Linux kernel internally goes FOUR TIMES
through the array supplied by poll(), so, it will perform better
than select() only with relatively small number of descriptors.
In other hand, as I've said, poll() can handle more than 1024 file
descriptors in a convenient way.

Actually, it is the best thing in the ST library to optimize.
Neither poll() nor select() will give the desirable performance
on typical server systems, because of large overhead. Suppose,
you have 2000 clients with only 10 residing on the fast
connections. It is clear, that poll() or select() will scan
the whole table only to determine that the same small set of
descriptors is active. It will require a completely
another approach to overcome this table scans on every call
to poll() or select(). This approach is to report the user
process the CHANGE in file descriptors state.

I encourage you to look at kqueue()/kevent() calls on
FreeBSD, the /dev/poll mechanism on Solaris,
/dev/poll and /dev/epoll on modern Linux systems.


-- 
Lev Walkin
vl...@ne...

[ST-devel] revision 1 of rpms&debs

[Wesley W. T. <we...@te...>]

Alright!

I've finished my first attempt to get libst fully packaged as an rpm and
deb. This source tree is based off of st-1.2.tar.gz.

The tarball is at: http://pez.ca/terpstra/libst-1.2.1.tar.gz
It is about 230k so I have not attached it.

To build debs: (as any user)
tar xvzf libst-1.2.1.tar.gz
cd libst-1.2.1
debuild

-- you will then find .deb packages in the parent directory

To build rpms: (as root unless you've already setup rpm)
rpm -ta libst-1.2.1.tar.gz

-- rpms will then be in /usr/src/redhat/ (or elsewhere if you have it
configured for non-root building)

What remains to be done:

I am fairly certain that I have messed it up for other architectures than
i686/linux/debian. 

I made several changes to the source which may be bad and need review. 
They were mostly aimed at getting autoconf sufficiently happy with
cross-compiling and automatic building.

The sox package still has a /usr/lib/libst.a which conflicts with
libst-dev_1.2.1-1_i386.deb.

I don't think I have the upstream maintainer correct. Who is the official
upstream maintainer?

Well, that's all!

-- 
Wesley W. Terpstra <we...@te...>

Re: [ST-devel] autoconf and libst

[Wesley W. T. <we...@te...>]

On Tue, Dec 25, 2001 at 02:50:46PM -0800, Lev Walkin wrote:
> >MD_USE_SYSV/BSD_ANON_MMAP:
> >	Idea 1:
> >#ifdef MAP_ANON
> >do BSD stuff
> >#els
> >do SYSV stuff
> >#endif
> >
> >	Idea 2:
> >always open /dev/zero
> >#ifdef MAP_ANON
> >flags |= MAP_ANON;
> >#endif
> >mmap
> 
> It is a bad idea to invoke useless open() call with mmap(), if we
> can avoid it.

Ok, that doesn't rule out idea 1 though. Does MAP_ANON being defined
guarantee it is available and works?

Alternately, if build system == host system: run a check for sysv/bsd
semantics. If cross compiling fall back to idea 2 or malloc. (whichever is
better)

Again, what's wrong with malloc? I thought it uses mmap for large
allocations internally?

> >MD_ACCEPT_NB_INHERITED:
> >
> >	Why not just assume it's not inherited and always reset the thing
> >nonblocking? Is it really such a performance hit?
> 
> System call is always a performance hit. But related with other
> types of activities during thread initialization, it is almost nothing.

Ok,

io.c:
static int st_nb_inherited = -1;

void st_determine_if_nb_inherit(int fd)
{
	long flags = fcntl(fd, F_GETFL);
	if ((flags & O_NONBLOCK) != 0)
		st_nb_inherited = 1;
	else
		st_nb_inherited = 0;
}

...
// code using #ifdef now:
if (st_nb_inherited == -1) st_derermine_if_nb_inherit(sock);
if (st_nb_inherited) _st_newfd_new(sock, 0, 1);
else                 _st_newfd_new(sock, 1, 1);
...
EOF

This should address the performance issue...?

> >MD_ALWAYS_UNSERIALIZED_ACCEPT:
> >
> >	I don't know enough about this to write a test. Could someone give
> >me a hand?
> 
> Actually, it is somewhat hard to write a multiprocess test of such
> a nature within automake framework. You should consider using internal
> __OS_Type__ variables in header or code files to protect this place.

Hrm. Ok then, which OSs need serialized accept?

One last thing: I have it using poll() whenever the function is available
since I am under the belief that poll() is faster than select() - is this
wrong?

Thanks!

-- 
Wesley W. Terpstra <we...@te...>

Re: [ST-devel] autoconf and libst

[Lev W. <vl...@ne...>]

> MD_STACK_GROWS_UP/DOWN:
> 
> 	I can't see how to detect this during configure without breaking
> cross-compilation. To work with cross-compiling you'd have to somehow fail
> to compile if it is STACK_GROWS_DOWN. It's easy enough to detect this at
> runtime anyways, and it's only in one file: stk.c and called at thread
> creation time. Hence I assume the cost to speed is not important?


Thread creation is very complicated task relative to other st
activities. Run-time dectection seems to not made things noticeably
worse.

> MD_USE_SYSV/BSD_ANON_MMAP:
> 
> 	I have a couple ideas how to deal with this.
> 	Idea 1:
> #ifdef MAP_ANON
> do BSD stuff
> #els
> do SYSV stuff
> #endif
> 
> 	Idea 2:
> always open /dev/zero
> #ifdef MAP_ANON
> flags |= MAP_ANON;
> #endif
> mmap


It is a bad idea to invoke useless open() call with mmap(), if we
can avoid it.

> 	I don't know enough about mmap to know whether these would work
> however. -- PS. What's wrong with malloc?



> MD_ACCEPT_NB_INHERITED:
> 
> 	Why not just assume it's not inherited and always reset the thing
> nonblocking? Is it really such a performance hit?


System call is always a performance hit. But related with other
types of activities during thread initialization, it is almost nothing.


> MD_ALWAYS_UNSERIALIZED_ACCEPT:
> 
> 	I don't know enough about this to write a test. Could someone give
> me a hand?


Actually, it is somewhat hard to write a multiprocess test of such
a nature within automake framework. You should consider using internal
__OS_Type__ variables in header or code files to protect this place.


> MD_SETJMP/MD_LONGJMP:
> 	
> 	I am presently running this order of tests:
> 	if __ia64__ defined, use _ia64_cxt_* and include the assembly file.
> 	if setjmp available, use it
> 	if _setjmp available, use it
> 	fail with an error
> 
> 	PS what's wrong with setjmp/longjmp for ia64?
> 
> MD_GET_UTIME:
> 
> 	Test for these functions:
> 	read_real_time time_base_to_time gettimeofday
> 	syssgi clock_gettime gethrtime ftime
> 
> 	Use them in this order:
> 	gettimeofday
> 	read_real_time & time_base_to_time
> 	syssgi & clock_gettime
> 	gethrtime
> 	ftime
> 	fail with an error
> 
> MD_INIT_CONTEXT:
> 
> 	I have no idea how to check for this one. If I were willing to give
> up cross-compilation I would simply write a test which tried each and use
> the one which doesn't crash. :-) However, I would rather not... The other
> option would be to examine the target machine's name and have a big lookup
> table for each machine. I am guessing this define is why the current 'make
> target' exists.
> 
> So, to summarize: I have it mostly autoconf/automake/libtool/rpm/deb'd up.
> As soon as I get some idea as to how to tackle MD_INIT_CONTEXT and the
> SERIALIZED_ACCEPT, I'll put together a tar.gz to be perused by you folks.
> 
> If I don't hear back from anybody, I'm going to assume that the order I
> check for things will work, always use serialized accept, and use a lookup
> based approach to MD_INIT_CONTEXT.
> 
> Hope to hear back from you guys!
> 
> 



-- 
Lev Walkin
vl...@ne...

[ST-devel] autoconf and libst

[Wesley W. T. <we...@te...>]

Ok, I've got most of the autoconf stuff going now. However, I had to make
some source level changes to make it work and there are a few things still
remaining:

From what I can see, md.h sets several macros based on the machine. This is
the main thing that needs to be setup to be auto-detected by autoconf.

MD_STACK_GROWS_UP/DOWN:

	I can't see how to detect this during configure without breaking
cross-compilation. To work with cross-compiling you'd have to somehow fail
to compile if it is STACK_GROWS_DOWN. It's easy enough to detect this at
runtime anyways, and it's only in one file: stk.c and called at thread
creation time. Hence I assume the cost to speed is not important?

MD_USE_SYSV/BSD_ANON_MMAP:

	I have a couple ideas how to deal with this.
	Idea 1:
#ifdef MAP_ANON
do BSD stuff
#els
do SYSV stuff
#endif

	Idea 2:
always open /dev/zero
#ifdef MAP_ANON
flags |= MAP_ANON;
#endif
mmap

	I don't know enough about mmap to know whether these would work
however. -- PS. What's wrong with malloc?

MD_ACCEPT_NB_INHERITED:

	Why not just assume it's not inherited and always reset the thing
nonblocking? Is it really such a performance hit?

MD_ALWAYS_UNSERIALIZED_ACCEPT:

	I don't know enough about this to write a test. Could someone give
me a hand?

MD_HAVE_SOCKLEN_T:

	No problem for autoconf

MD_SETJMP/MD_LONGJMP:
	
	I am presently running this order of tests:
	if __ia64__ defined, use _ia64_cxt_* and include the assembly file.
	if setjmp available, use it
	if _setjmp available, use it
	fail with an error

	PS what's wrong with setjmp/longjmp for ia64?

MD_GET_UTIME:

	Test for these functions:
	read_real_time time_base_to_time gettimeofday
	syssgi clock_gettime gethrtime ftime

	Use them in this order:
	gettimeofday
	read_real_time & time_base_to_time
	syssgi & clock_gettime
	gethrtime
	ftime
	fail with an error

MD_INIT_CONTEXT:

	I have no idea how to check for this one. If I were willing to give
up cross-compilation I would simply write a test which tried each and use
the one which doesn't crash. :-) However, I would rather not... The other
option would be to examine the target machine's name and have a big lookup
table for each machine. I am guessing this define is why the current 'make
target' exists.

So, to summarize: I have it mostly autoconf/automake/libtool/rpm/deb'd up.
As soon as I get some idea as to how to tackle MD_INIT_CONTEXT and the
SERIALIZED_ACCEPT, I'll put together a tar.gz to be perused by you folks.

If I don't hear back from anybody, I'm going to assume that the order I
check for things will work, always use serialized accept, and use a lookup
based approach to MD_INIT_CONTEXT.

Hope to hear back from you guys!

-- 
Wesley W. Terpstra <we...@te...>

[ST-devel] packaging libst: autoconf and libtool

[Wesley W. T. <we...@te...>]

Heya! 

I've been looking into packaging libst as a .deb and .rpm. I have versions
that work on linux, but they assume that libtool is installed and that it
should use -DLINUX.

For it to build properly on any target platform, I need to make a configure
script that will detect those settings that libst needs. Right now, the
system of 'make arch' really doesn't cut it for cross-platform packaging.

Has anyone already looked into making an autoconf configure script?
From what I can tell, the only things that would need to be detected are:
USE_POLL
MD_ALWAYS_UNSERIALIZED_ACCEPT
MD_STACK_GROWS_DOWN/UP
MD_ACCEPT_NB_(NOT_)INHERITED
MALLOC_STACK
MD_USE_SYSV_ANON_MMAP
MD_GET_UTIME
MD_USE_SYSV_ANON_MMAP
and the presence of a few headers. This is certainly doable by autoconf, but
I have no desire to duplicate someone else's work. :-)

If anyone has already written portable tests for the context switching
MD_STACK_GROWS_DOWN/UP, MALLOC_STACK, and MD_ACCEPT_NB_(NOT_)INHERITED
stuff, I'd love to have them. The rest I'm sure I'll get right, but the ones
above I have no way to test on all the target platforms.

The way I figure autoconf should approach the issue is to try various
context switching mechanisms in order of which is faster and select the
first which works. Detect whether poll or anonymous mmap are available, etc
and then output a config.h with the appropriate defines. In this manner, it
will work on any platform which provides some combination of the necessary
features.

The ia64 code -- what is it for? Is it faster than the setjmp/longjmp
technique? Or is setjmp/longjmp not available on this platform? (I am not
familiar with the ia64)

Also, the method used to build linux libraries presently employed is
actually incorrect... You should compile the shared objects with -fPIC -FPIC
and when linking specify the soname of the output library. Otherwise the
library will not be relocatable, and programs will link to libst.so instead
of libst.so.1 as they should.

To correct this, in my debian packaging of the project I have it use libtool
to compile the project. Libtool is very good at determining the correct
options to use for compiling on a target platform. It knows more than any
single developer I've met. :-)

However, if I do make a patch which uses libtool and autoconf, the tar.gz
and build time will be significantly increased. I'm not sure whether
download size / build time is a goal of the project? It won't increase the
compiled size of the library -- just the source tar.gz.

It would likely be 180k instead of 70k. This is because typically libtool
and autoconf are included in the source tar.gz so that if the user doesn't
have them, the build still works. The other alternative is to require the
user to have these packages installed in which case there will be almost no
size impact.

Further, libtool and configure will end up doing a lot of tests that take
some time. Perhaps it's overkill given the size of the project, but it is
how most OS projects deal with it. :-)

Finally, do you have access to the platforms on which this presently builds?
If I do get it working with autoconf/libtool it would be nice if I could
test it on all the platforms it worked on before.

One last thing -- the program 'sox' has a /usr/lib/libst.a in it already
which conflicts with the obvious installed library name. Not sure what to do
about this. Maybe ask them to rename their supporting lib? ;-)

Thanks! I want to contribute to this project (which I love using), but I
don't know who's involved and what's been done in the direction of packaging
it already, etc.

PS. I'm out in the middle of nowhere using a modem for Christmas so my email
response times will be a bit longer than normal.

-- 
Wesley W. Terpstra <we...@te...>

[ST-devel] packaging libst: autoconf and libtool

[Wesley W. T. <we...@io...>]

Heya! 

I've been looking into packaging libst as a .deb and .rpm. I have versions
that work on linux, but they assume that libtool is installed and that it
should use -DLINUX.

For it to build properly on any target platform, I need to make a configure
script that will detect those settings that libst needs. Right now, the
system of 'make arch' really doesn't cut it for cross-platform packaging.

Has anyone already looked into making an autoconf configure script?
From what I can tell, the only things that would need to be detected are:
USE_POLL
MD_ALWAYS_UNSERIALIZED_ACCEPT
MD_STACK_GROWS_DOWN/UP
MD_ACCEPT_NB_(NOT_)INHERITED
MALLOC_STACK
MD_USE_SYSV_ANON_MMAP
MD_GET_UTIME
MD_USE_SYSV_ANON_MMAP
and the presence of a few headers. This is certainly doable by autoconf, but
I have no desire to duplicate someone else's work. :-)

If anyone has already written portable tests for the context switching
MD_STACK_GROWS_DOWN/UP, MALLOC_STACK, and MD_ACCEPT_NB_(NOT_)INHERITED
stuff, I'd love to have them. The rest I'm sure I'll get right, but the ones
above I have no way to test on all the target platforms.

The way I figure autoconf should approach the issue is to try various
context switching mechanisms in order of which is faster and select the
first which works. Detect whether poll or anonymous mmap are available, etc
and then output a config.h with the appropriate defines. In this manner, it
will work on any platform which provides some combination of the necessary
features.

The ia64 code -- what is it for? Is it faster than the setjmp/longjmp
technique? Or is setjmp/longjmp not available on this platform? (I am not
familiar with the ia64)

Also, the method used to build linux libraries presently employed is
actually incorrect... You should compile the shared objects with -fPIC -FPIC
and when linking specify the soname of the output library. Otherwise the
library will not be relocatable, and programs will link to libst.so instead
of libst.so.1 as they should.

To correct this, in my debian packaging of the project I have it use libtool
to compile the project. Libtool is very good at determining the correct
options to use for compiling on a target platform. It knows more than any
single developer I've met. :-)

However, if I do make a patch which uses libtool and autoconf, the tar.gz
and build time will be significantly increased. I'm not sure whether
download size / build time is a goal of the project? It won't increase the
compiled size of the library -- just the source tar.gz.

It would likely be 180k instead of 70k. This is because typically libtool
and autoconf are included in the source tar.gz so that if the user doesn't
have them, the build still works. The other alternative is to require the
user to have these packages installed in which case there will be almost no
size impact.

Further, libtool does and configure will end up doing a lot of tests that
take some time. Perhaps it's overkill given the size of the project, but it
is how most OS projects deal with it. :-)

Finally, do you have access to the platforms on which this presently builds?
If I do get it working with autoconf/libtool it would be nice if I could
test it on all the platforms it worked on before.

One last thing -- the program 'sox' has a /usr/lib/libst.a in it already
which conflicts with the obvious installed library name. Not sure what to do
about this. Maybe ask them to rename their supporting lib? ;-)

Thanks! I want to contribute to this project (which I love using), but I
don't know who's involved and what's been done in the direction of packaging
it already, etc.

PS. I'm out in the middle of nowhere using a modem for Christmas so my email
response times will be a bit longer than normal.

-- 
Wesley W. Terpstra <we...@te...>

[ST-devel] 1.3c

[Mike A. <mj...@ho...>]

Experimental version 1.3c of the State Threads Library is now available.
It adds some debugging support, an important optimization, and more
documentation to 1.3b.
	http://state-threads.sourceforge.net/

If someone other than Gene and me verifies that this release is stable
I'll release it as 1.3, oh, after the holidays or so.
-- 
Mike Abbott    ma...@us...
State Threads Project co-administrator

[ST-devel] 1.3b

[Mike A. <mj...@ho...>]

Experimental version 1.3b of the State Threads Library is now available.
It differs from 1.3a only in documentation and comments.  I added a
description of the proper use of timeouts to the Programming Notes,
added Gene's explanation to st_connect(), updated copyrights, and
changed the link to the project's site.
	http://state-threads.sourceforge.net/

So far I have received no feedback on the *_resid() additions in 1.3a.
Would someone review and test them please?  Thanks.
-- 
Mike Abbott    ma...@us...
State Threads Project co-administrator