Menu
▾
▴
syllable-developer
|
From: Stefano D'A. <zan...@gm...> - 2008-10-24 10:43:24
|
Hi all, I'm trying to port my DSP framework to Syllable, or at least I'd like to see it compiled and working on Syllable now and maybe write better support in the future (both in the sense of build system/installation and code). So, my first problem is: I have the following line of code somewhere: err = pthread_mutexattr_settype(&mutexattr, PTHREAD_MUTEX_ERRORCHECK); which returns a non-0 value (probably EINVAL). Does this mean that Syllable does not recognize PTHREAD_MUTEX_ERRORCHECK as a valid POSIX mutex type or is there something else I should know? Thanks, Stefano |
|
From: Kristian V. D. V. <kri...@qu...> - 2008-10-24 12:28:59
|
On Fri, 2008-10-24 at 11:41 +0100, Stefano D'Angelo wrote:
> So, my first problem is: I have the following line of code somewhere:
>
> err = pthread_mutexattr_settype(&mutexattr, PTHREAD_MUTEX_ERRORCHECK);
>
> which returns a non-0 value (probably EINVAL). Does this mean that
> Syllable does not recognize PTHREAD_MUTEX_ERRORCHECK as a valid POSIX
> mutex type or is there something else I should know?
It's a stupid bug in pthread_mutexattr_settype. Can you spot the problem?
if( ( type != PTHREAD_MUTEX_DEFAULT ) ||
( type != PTHREAD_MUTEX_ERRORCHECK ) ||
( type != PTHREAD_MUTEX_NORMAL ) ||
( type != PTHREAD_MUTEX_RECURSIVE ) )
return( EINVAL );
Hmm, no I don't think that's quite right...
I'll fix it in CVS later and email you a new libpthread if you like?
--
Vanders
http://www.syllable.org
|
|
From: Kristian V. D. V. <va...@li...> - 2008-10-24 19:33:41
Attachments:
libpthread.7z
|
On Fri, 2008-10-24 at 13:31 +0100, Kristian Van Der Vliet wrote: > I'll fix it in CVS later and email you a new libpthread if you like? Here it is. Un-7zip it and put the file in /system/libraries. You may need to reboot to force the new library to be loaded. I also found and fixed at least two other places where I'd made the same schoolboy error. -- Vanders http://www.syllable.org |
|
From: Stefano D'A. <zan...@gm...> - 2008-10-26 03:16:12
|
2008/10/24 Kristian Van Der Vliet <va...@li...>: > On Fri, 2008-10-24 at 13:31 +0100, Kristian Van Der Vliet wrote: >> I'll fix it in CVS later and email you a new libpthread if you like? > > Here it is. Un-7zip it and put the file in /system/libraries. You may > need to reboot to force the new library to be loaded. > > I also found and fixed at least two other places where I'd made the same > schoolboy error. Ok, with the fixed library we go from 8/9 tests failed to 2/9 failures. I will investigate on the remaining failures, in the meanwhile I noticed a couple of things: - there seems to be no ldd script in Syllable - is this intentional? - the library you gave is name libpthread.so.2 while /system/libraries/libpthread.so linked to /system/libraries/libpthread.so.1 (using Syllable 0.6.5) - I guess this implies that my stuff will only work with Syllable >= 0.6.6? Stefano |
|
From: Stefano D'A. <zan...@gm...> - 2008-10-26 04:35:34
|
One of the two remaining problems goes like this: * the main thread allocates a mutex (ok) * another thread locks it (ok) * the same thread which locked it tries to unlock it, but pthread_mutex_unlock() returns EPERM The mutex type is PTHREAD_MUTEX_ERRCHECK and it seems to always work this way. The other problem is probably related to libdl, I will try to find out more later. Plus vi shows strange behaviour with home/end keys on Syllable (using IT keyboard)... is that something configurable or what? Stefano |
|
From: Stefano D'A. <zan...@gm...> - 2008-10-26 05:51:43
|
The last problem is a fault of mine... however I noticed that dlsym() sets errno to 131 when not finding some symbol (using strerror I get "Unknown error - 131", but I see it is ENOSYM in errno.h), which is probably fine since the POSIX standard doesn't seem to even mention errno is the dlsym page... however, in case you are interested in that, all other OSes I tried probably don't let dlsym set errno (Windows/MSYS, Linux, BSDs), while the AIX man page for dlsym states: "If the named symbol is found, its address is returned. If the named symbol is not found, NULL is returned and errno is set to 0." Just to let you know ;-) Stefano |
|
From: Kristian V. D. V. <va...@li...> - 2008-10-26 12:48:02
|
On Sun, 2008-10-26 at 04:16 +0100, Stefano D'Angelo wrote: > 2008/10/24 Kristian Van Der Vliet <va...@li...>: > > On Fri, 2008-10-24 at 13:31 +0100, Kristian Van Der Vliet wrote: > >> I'll fix it in CVS later and email you a new libpthread if you like? > > I will investigate on the remaining failures, in the meanwhile I > noticed a couple of things: > > - there seems to be no ldd script in Syllable - is this intentional? Yes. The ldd script that is part of binutils uses ld.so, which we do not have. Use readelf -d instead and look at the NEEDED items. The output isn't as clear as ldd but shared libraries on Syllable rarely have complex dependency trees. > - the library you gave is name libpthread.so.2 while > /system/libraries/libpthread.so linked to > /system/libraries/libpthread.so.1 (using Syllable 0.6.5) - I guess > this implies that my stuff will only work with Syllable >= 0.6.6? Yes, but we do ship an ABI compatible libpthread.so.1 and I could & should apply the same fix there. > One of the two remaining problems goes like this: > > * the main thread allocates a mutex (ok) > * another thread locks it (ok) > * the same thread which locked it tries to unlock it, but > pthread_mutex_unlock() returns EPERM > > The mutex type is PTHREAD_MUTEX_ERRCHECK and it seems to always work > this way. Thanks, I'll look into it. Does it make any difference if the mutex is not MUTEX_ERRCHECK? > The last problem is a fault of mine... however I noticed that dlsym() > sets errno to 131 when not finding some symbol (using strerror I get > "Unknown error - 131", but I see it is ENOSYM in errno.h), which is > probably fine since the POSIX standard doesn't seem to even mention > errno is the dlsym page... however, in case you are interested in > that, all other OSes I tried probably don't let dlsym set errno > (Windows/MSYS, Linux, BSDs), while the AIX man page for dlsym states: > > "If the named symbol is found, its address is returned. If the named > symbol is not found, NULL is returned and errno is set to 0." Hmm, that's what it should be doing, but libdl does use the native call get_symbol_address() which *may* set errno. libdl itself doesn't touch errno, so it wont clear errno to zero if get_symbol_address() fails. -- Vanders http://www.syllable.org |
|
From: Stefano D'A. <zan...@gm...> - 2008-10-27 09:31:20
|
2008/10/26 Kristian Van Der Vliet <va...@li...>: >> - the library you gave is name libpthread.so.2 while >> /system/libraries/libpthread.so linked to >> /system/libraries/libpthread.so.1 (using Syllable 0.6.5) - I guess >> this implies that my stuff will only work with Syllable >= 0.6.6? > > Yes, but we do ship an ABI compatible libpthread.so.1 and I could & > should apply the same fix there. I don't really care, the only thing I care about is the "minimum requirements" thing (so 0.6.6 should be it anyway) ;-) >> One of the two remaining problems goes like this: >> >> * the main thread allocates a mutex (ok) >> * another thread locks it (ok) >> * the same thread which locked it tries to unlock it, but >> pthread_mutex_unlock() returns EPERM >> >> The mutex type is PTHREAD_MUTEX_ERRCHECK and it seems to always work >> this way. > > Thanks, I'll look into it. Does it make any difference if the mutex is > not MUTEX_ERRCHECK? I tried without setting mutex type and it works correctly, but... another problem came out: pthread_join() occasionally returning ESRCH. Now, these tests work properly at least on Linux, so I'm pretty sure that the code is valid. In this case the test launches one hundred threads, lets them do some concurrent insertion on a list, then waits for each one of them using pthread_join(). pthread_join() is called inside a for loop which scans the array of threads. The ESRCH return happens usually for the second or third thread trying to be joined, so it does look like some kind of bug. Now, I understand these kind of stuff is quite hard to debug, so if you want to reproduce these bugs on your own I can give you detailed instructions on how to obtain and build the code I'm working on, otherwise I can just keep up pinging you whenever I have problems. You choose ;-) >> The last problem is a fault of mine... however I noticed that dlsym() >> sets errno to 131 when not finding some symbol (using strerror I get >> "Unknown error - 131", but I see it is ENOSYM in errno.h), which is >> probably fine since the POSIX standard doesn't seem to even mention >> errno is the dlsym page... however, in case you are interested in >> that, all other OSes I tried probably don't let dlsym set errno >> (Windows/MSYS, Linux, BSDs), while the AIX man page for dlsym states: >> >> "If the named symbol is found, its address is returned. If the named >> symbol is not found, NULL is returned and errno is set to 0." > > Hmm, that's what it should be doing, but libdl does use the native call > get_symbol_address() which *may* set errno. libdl itself doesn't touch > errno, so it wont clear errno to zero if get_symbol_address() fails. As said, I think your way of doing things is perfectly valid and POSIX-compatible, so I don't have problems with that. However some buggy software could rely on this kind of behaviour, so it's up to you whether you want to support those too. Just trying to help :-) Stefano |
|
From: Stefano D'A. <zan...@gm...> - 2008-10-27 09:28:36
|
I was forgetting... other than those problems, the synchronization test works fine, which is very likely to indicate that mutexes, at least, work properly :-) Stefano |
|
From: Stefano D'A. <zan...@gm...> - 2008-11-01 10:42:14
|
I investigated a bit into Syllable's source code and it seems like
some weird stuff is going on... First, in pthread_exit(), this cycle
seems to be broken:
231 while ( cleanup )
232 {
233 if ( cleanup->routine )
234 (*cleanup->routine) (cleanup->arg);
235 cleanup = cleanup->prev;
236 free( cleanup );
237 }
since after the first loop, cleanup points to a freed memory location,
which shouldn't be considered valid any more (unless you have very
very strange memory handling routines/conventions, but I doubt that).
At first glance I'd say it should be more like this:
while (cleanup)
{
if (cleanup->routine)
(*cleanup->routine) (cleanup->arg);
tmp = cleanup;
cleanup = cleanup->prev;
free(tmp);
}
Going back to the original problem, it seems like pthread_exit()
completely destroys the thread, which is not a POSIX-compliant
behaviour, since it won't be found by pthread_join()s happening after
the thread has terminated.
This would be ok for PTHREAD_CREATE_DETACHED threads, but the POSIX
standards states that the default detachstate is
PTHREAD_CREATE_JOINABLE.
Stefano
|
|
From: Kristian V. D. V. <va...@li...> - 2008-11-01 12:18:56
|
On Sat, 2008-11-01 at 11:42 +0100, Stefano D'Angelo wrote:
> I investigated a bit into Syllable's source code and it seems like
> some weird stuff is going on... First, in pthread_exit(), this cycle
> seems to be broken:
>
> 231 while ( cleanup )
> 232 {
> 233 if ( cleanup->routine )
> 234 (*cleanup->routine) (cleanup->arg);
> 235 cleanup = cleanup->prev;
> 236 free( cleanup );
> 237 }
>
> since after the first loop, cleanup points to a freed memory location,
> which shouldn't be considered valid any more (unless you have very
> very strange memory handling routines/conventions, but I doubt that).
Believe it or not that loop is actually O.K. It works backwards through
the list, calling the cleanup handlers and then freeing the structures
as it goes. For the first item in the list, prev will be NULL and
free(NULL) is a valid no-op, and the loop will then exit. I admit it is
non-obvious.
> Going back to the original problem, it seems like pthread_exit()
> completely destroys the thread, which is not a POSIX-compliant
> behaviour, since it won't be found by pthread_join()s happening after
> the thread has terminated.
>
> This would be ok for PTHREAD_CREATE_DETACHED threads, but the POSIX
> standards states that the default detachstate is
> PTHREAD_CREATE_JOINABLE.
The OpenGroup spec says:
"The pthread_join() function suspends execution of the calling thread
until the target thread terminates, unless the target thread has already
terminated. On return from a successful pthread_join() call with a
non-NULL value_ptr argument, the value passed to pthread_exit() by the
terminating thread is made available in the location referenced by
value_ptr. When a pthread_join() returns successfully, the target thread
has been terminated. The results of multiple simultaneous calls to
pthread_join() specifying the same target thread are undefined. If the
thread calling pthread_join() is canceled, then the target thread will
not be detached.
[ESRCH]
No thread could be found corresponding to that specified by the
given thread ID."
which is what the current implementation does: if pthread_join() is
called on a thread that has already called pthread_exit() it returns
ESRCH. However it seems I have been tripped up by "When a pthread_join()
returns successfully, the target thread has been terminated."! Reading
that now my interpretation is that pthread_join() should return
immediately when called on a thread that has exited, rather than
returning ESRCH.
I *think* there is a way to fix this, sort of: wait_for_thread() will
return -ECHILD if the target thread does not exist, which can be
interpreted as "Thread has exited". However there is then no way to
detect cases where the thread ID is invalid I.e. to return ESRCH, but
this would seem to be a less important edge case.
--
Vanders
http://www.syllable.org
|
|
From: Stefano D'A. <zan...@gm...> - 2008-11-01 13:44:14
|
2008/11/1 Kristian Van Der Vliet <va...@li...>:
> On Sat, 2008-11-01 at 11:42 +0100, Stefano D'Angelo wrote:
>> I investigated a bit into Syllable's source code and it seems like
>> some weird stuff is going on... First, in pthread_exit(), this cycle
>> seems to be broken:
>>
>> 231 while ( cleanup )
>> 232 {
>> 233 if ( cleanup->routine )
>> 234 (*cleanup->routine) (cleanup->arg);
>> 235 cleanup = cleanup->prev;
>> 236 free( cleanup );
>> 237 }
>>
>> since after the first loop, cleanup points to a freed memory location,
>> which shouldn't be considered valid any more (unless you have very
>> very strange memory handling routines/conventions, but I doubt that).
>
> Believe it or not that loop is actually O.K. It works backwards through
> the list, calling the cleanup handlers and then freeing the structures
> as it goes. For the first item in the list, prev will be NULL and
> free(NULL) is a valid no-op, and the loop will then exit. I admit it is
> non-obvious.
>
>> Going back to the original problem, it seems like pthread_exit()
>> completely destroys the thread, which is not a POSIX-compliant
>> behaviour, since it won't be found by pthread_join()s happening after
>> the thread has terminated.
>>
>> This would be ok for PTHREAD_CREATE_DETACHED threads, but the POSIX
>> standards states that the default detachstate is
>> PTHREAD_CREATE_JOINABLE.
>
> The OpenGroup spec says:
>
> "The pthread_join() function suspends execution of the calling thread
> until the target thread terminates, unless the target thread has already
> terminated. On return from a successful pthread_join() call with a
> non-NULL value_ptr argument, the value passed to pthread_exit() by the
> terminating thread is made available in the location referenced by
> value_ptr. When a pthread_join() returns successfully, the target thread
> has been terminated. The results of multiple simultaneous calls to
> pthread_join() specifying the same target thread are undefined. If the
> thread calling pthread_join() is canceled, then the target thread will
> not be detached.
>
> [ESRCH]
> No thread could be found corresponding to that specified by the
> given thread ID."
>
> which is what the current implementation does: if pthread_join() is
> called on a thread that has already called pthread_exit() it returns
> ESRCH. However it seems I have been tripped up by "When a pthread_join()
> returns successfully, the target thread has been terminated."! Reading
> that now my interpretation is that pthread_join() should return
> immediately when called on a thread that has exited, rather than
> returning ESRCH.
>
> I *think* there is a way to fix this, sort of: wait_for_thread() will
> return -ECHILD if the target thread does not exist, which can be
> interpreted as "Thread has exited". However there is then no way to
> detect cases where the thread ID is invalid I.e. to return ESRCH, but
> this would seem to be a less important edge case.
Well... it seems like the POSIX standard is a bit ambiguous (as usual
:-P), but looking around I've always seen it implemented "the other
way": pthread_join() will return 0 on terminated threads (that's for
sure on Linux, FreeBSD, DragonFlyBSD and Haiku).
However, it is my understanding that the original meaning of the
standard is that you can have two types of threads:
* PTHREAD_CREATE_DETACHED threads, which you just can't join or detach
(thus can't get their return value) - those threads should be
completely destroyed when they call pthread_exit();
* PTHREAD_CREATE_JOINABLE threads (the default), which the system
should keep their return value (hence reference to them) in memory
until pthread_join() is called.
I don't think any other interpration, even if valid according to the
"literal standard", makes much sense.
Don't misunderstand, I'm not trying to dictate what you should do, I'm
just trying to help you (and me, porting that stuff).
Stefano
|
|
From: Kristian V. D. V. <va...@li...> - 2008-11-01 13:58:45
|
On Sat, 2008-11-01 at 14:44 +0100, Stefano D'Angelo wrote:
> 2008/11/1 Kristian Van Der Vliet <va...@li...>:
> > On Sat, 2008-11-01 at 11:42 +0100, Stefano D'Angelo wrote:
> >> I investigated a bit into Syllable's source code and it seems like
> >> some weird stuff is going on... First, in pthread_exit(), this cycle
> >> seems to be broken:
> >>
> >> 231 while ( cleanup )
> >> 232 {
> >> 233 if ( cleanup->routine )
> >> 234 (*cleanup->routine) (cleanup->arg);
> >> 235 cleanup = cleanup->prev;
> >> 236 free( cleanup );
> >> 237 }
> >>
> >> since after the first loop, cleanup points to a freed memory location,
> >> which shouldn't be considered valid any more (unless you have very
> >> very strange memory handling routines/conventions, but I doubt that).
> >
> > Believe it or not that loop is actually O.K. It works backwards through
> > the list, calling the cleanup handlers and then freeing the structures
> > as it goes. For the first item in the list, prev will be NULL and
> > free(NULL) is a valid no-op, and the loop will then exit. I admit it is
> > non-obvious.
Anthony has pointed out to me that there is an issue here in that the
first cleanup structure in the list is never free'd, which is a valid
problem here.
>> Going back to the original problem, it seems like pthread_exit()
> >> completely destroys the thread, which is not a POSIX-compliant
> >> behaviour, since it won't be found by pthread_join()s happening after
> >> the thread has terminated.
> >>
> >> This would be ok for PTHREAD_CREATE_DETACHED threads, but the POSIX
> >> standards states that the default detachstate is
> >> PTHREAD_CREATE_JOINABLE.
> >
> > The OpenGroup spec says:
> >
> > "The pthread_join() function suspends execution of the calling thread
> > until the target thread terminates, unless the target thread has already
> > terminated. On return from a successful pthread_join() call with a
> > non-NULL value_ptr argument, the value passed to pthread_exit() by the
> > terminating thread is made available in the location referenced by
> > value_ptr. When a pthread_join() returns successfully, the target thread
> > has been terminated. The results of multiple simultaneous calls to
> > pthread_join() specifying the same target thread are undefined. If the
> > thread calling pthread_join() is canceled, then the target thread will
> > not be detached.
> >
> > [ESRCH]
> > No thread could be found corresponding to that specified by the
> > given thread ID."
> >
> > which is what the current implementation does: if pthread_join() is
> > called on a thread that has already called pthread_exit() it returns
> > ESRCH. However it seems I have been tripped up by "When a pthread_join()
> > returns successfully, the target thread has been terminated."! Reading
> > that now my interpretation is that pthread_join() should return
> > immediately when called on a thread that has exited, rather than
> > returning ESRCH.
> >
> > I *think* there is a way to fix this, sort of: wait_for_thread() will
> > return -ECHILD if the target thread does not exist, which can be
> > interpreted as "Thread has exited". However there is then no way to
> > detect cases where the thread ID is invalid I.e. to return ESRCH, but
> > this would seem to be a less important edge case.
>
> Well... it seems like the POSIX standard is a bit ambiguous (as usual
> :-P), but looking around I've always seen it implemented "the other
> way": pthread_join() will return 0 on terminated threads (that's for
> sure on Linux, FreeBSD, DragonFlyBSD and Haiku).
>
> However, it is my understanding that the original meaning of the
> standard is that you can have two types of threads:
> * PTHREAD_CREATE_DETACHED threads, which you just can't join or detach
> (thus can't get their return value) - those threads should be
> completely destroyed when they call pthread_exit();
> * PTHREAD_CREATE_JOINABLE threads (the default), which the system
> should keep their return value (hence reference to them) in memory
> until pthread_join() is called.
>
> I don't think any other interpration, even if valid according to the
> "literal standard", makes much sense.
No I think you're right, otherwise it's a potential race condition
between one thread exiting and the other calling join: the joining
thread could interpret the return from ESRCH as a fatal error and abort,
when it isn't.
> Don't misunderstand, I'm not trying to dictate what you should do, I'm
> just trying to help you (and me, porting that stuff).
No, this is very helpful. It's nice to have some feedback on this sort
of level. Without it I can't improve Syllable :)
--
Vanders
http://www.syllable.org
|
|
From: Stefano D'A. <zan...@gm...> - 2008-11-01 21:22:58
|
2008/11/1 Kristian Van Der Vliet <va...@li...>: > No, this is very helpful. It's nice to have some feedback on this sort > of level. Without it I can't improve Syllable :) Good to know you're open to feedback :-) Stefano |
|
From: Stephan A. <sup...@gm...> - 2008-11-01 14:39:10
|
Kristian Van Der Vliet wrote:
> On Sat, 2008-11-01 at 14:44 +0100, Stefano D'Angelo wrote:
> > 2008/11/1 Kristian Van Der Vliet <va...@li...>:
> > > On Sat, 2008-11-01 at 11:42 +0100, Stefano D'Angelo wrote:
> > >> I investigated a bit into Syllable's source code and it seems like
> > >> some weird stuff is going on... First, in pthread_exit(), this cycle
> > >> seems to be broken:
> > >>
> > >> 231 while ( cleanup )
> > >> 232 {
> > >> 233 if ( cleanup->routine )
> > >> 234 (*cleanup->routine) (cleanup->arg);
> > >> 235 cleanup = cleanup->prev;
> > >> 236 free( cleanup );
> > >> 237 }
> > >>
> > >> since after the first loop, cleanup points to a freed memory
> > >> location, which shouldn't be considered valid any more (unless you
> > >> have very very strange memory handling routines/conventions, but I
> > >> doubt that).
> > >
> > > Believe it or not that loop is actually O.K. It works backwards
> > > through the list, calling the cleanup handlers and then freeing the
> > > structures as it goes. For the first item in the list, prev will be
> > > NULL and free(NULL) is a valid no-op, and the loop will then exit. I
> > > admit it is non-obvious.
>
> Anthony has pointed out to me that there is an issue here in that the
> first cleanup structure in the list is never free'd, which is a valid
> problem here.
No, it is really more broken than that. After calling free(cleanup), the
pointer points to free()d memory, but you happly dereference it in the next
iteration of the loop.
Best regards,
-Stephan
|
|
From: Kristian V. D. V. <va...@li...> - 2008-11-01 15:57:31
|
On Sat, 2008-11-01 at 15:39 +0100, Stephan Assmus wrote:
> Kristian Van Der Vliet wrote:
> > On Sat, 2008-11-01 at 14:44 +0100, Stefano D'Angelo wrote:
> > > 2008/11/1 Kristian Van Der Vliet <va...@li...>:
> > > > On Sat, 2008-11-01 at 11:42 +0100, Stefano D'Angelo wrote:
> > > >> I investigated a bit into Syllable's source code and it seems like
> > > >> some weird stuff is going on... First, in pthread_exit(), this cycle
> > > >> seems to be broken:
> > > >>
> > > >> 231 while ( cleanup )
> > > >> 232 {
> > > >> 233 if ( cleanup->routine )
> > > >> 234 (*cleanup->routine) (cleanup->arg);
> > > >> 235 cleanup = cleanup->prev;
> > > >> 236 free( cleanup );
> > > >> 237 }
> > > >>
> > > >> since after the first loop, cleanup points to a freed memory
> > > >> location, which shouldn't be considered valid any more (unless you
> > > >> have very very strange memory handling routines/conventions, but I
> > > >> doubt that).
> > > >
> > > > Believe it or not that loop is actually O.K. It works backwards
> > > > through the list, calling the cleanup handlers and then freeing the
> > > > structures as it goes. For the first item in the list, prev will be
> > > > NULL and free(NULL) is a valid no-op, and the loop will then exit. I
> > > > admit it is non-obvious.
> >
> > Anthony has pointed out to me that there is an issue here in that the
> > first cleanup structure in the list is never free'd, which is a valid
> > problem here.
>
> No, it is really more broken than that. After calling free(cleanup), the
> pointer points to free()d memory, but you happly dereference it in the next
> iteration of the loop.
Yes, you're right. I had some sort of mental disconnect and couldn't see
the deference in the next iteration. Stefanos original solution is
correct.
--
Vanders
http://www.syllable.org
|
Thanks for helping keep SourceForge clean.
X