iamb-dev-internal — Discussion of the gory details involved in a rewrite for the next major version

Re: [Iamb-dev-internal] API, development, etc

[Matt C. <su...@qx...>]

i'll be drafting an rfc about all this, clarifying what i remember us 
deciding and what i'm hearing now. hopefully i'll ship that out tonight 
all going well (and my firewall coming back online, blasted piece of 
crap).  

---------
Matt Cashner
Web Application Developer
The Creative Group (http://www.cre8tivegroup.com)
su...@ea... | Codito, ergo sum.

Re: [Iamb-dev-internal] API, development, etc

[Ken R. <kr...@qx...>]

Matt Cashner wrote:
> 
> On Mon, 8 Jan 2001, Ken Rietz wrote:
> 
> > Check this out. We have to know how $stree is constructed in order
> > to manipulate it correctly. This algorithm (I think) creates the
> > $stree that Dan and I seemed to think best, except for putting
> > in initial "[" and final "]" to reduce the number of cases in the
> > adding items phase.
> 
> you had code for this (many moons ago), did you not?

Not for adding. The code I had was for parsing something already
of this form down into a hash that could be used for multiple
purposes. The question is whether or not we want to keep the $stree
around in memory, or the hash. The $stree is clearly what we
would want to store in the database, though, so even if we use
the hash, we will also need a way to reconstruct the stree from it.

> Also, let me get this on the record once and for all.  If this algorithm
> could be substantially improved by being either in C via Inline.pm or in
> XS, then do it.  Just keep an eye to porting while doing it.  Beyond that,
> do what you think needs to be done to make this as fast as possible but
> still maintainable.
> 
> > The first two would be simple: regexes would handle them well. The
> > problem of finding matching "[" and "]" using regexes is hard, and
> > inefficient to boot. However, it can be done by a straightforward
> > character-by-character scan. That would be my first try.
> 
> cpan> install Text::Balanced

The character-by-character scan in C might or might not be
faster than the Text::Balanced. The others would be as fast
or faster than C by using the (optimized) regex engine. This
opinion needs to be benchmarked. I volunteer for that, once
we get the API squared away.

> damian++

He the man :-)

> > Clearly, adding threads and articles is critical, so if this
> > passes muster, then I'll work on the rest of the API as well
> > as some of the code.
> 
> sounds good to me.

Actually, what I was hoping for was a close, and critical,
examination of the algorithm. If it is in any way flaky, we
are going to be in a world of hurt. So, if you would please
do that, I'd be grateful.

> > One consideration is that no ID number--be it forum, thread, or
> > article--can equal any other ID, or this whole process is doomed.
> 
> ok. forumIDs dont apply to strees as the strees are stored per thread not
> per forum.

Huh? This is different than I remember, but I no longer trust my
memory. I thought that each forum would have its own stree. If that
is not the case, I need to do a lot of re-evaluating.

In my thinking, each thread and subthread would have their own strees,
but the forum stree is the master one, the one that is stored. The
others would be derived from the forum stree on the fly. So all
API operations have to be able to work on the forum strees, though
they will be applied to any stree. I thought that such parsing
was going to be one of the main uses of the API, actually.

The alternative is to keep the forum stree on disk, parsed once
into the the memory hash from long ago. Stree operations would
be translated into hash operations in that case. The advantage
of the API is that we could change from a memory stree to a memory
hash implementation without disruption. That might be the only
way to benchmark the two against each other.

> [snip] and threads are just articles,
> yes?

That is my understanding as well.

> the threadID is just the ID of the article at the head of the
> Thread.

Also my understanding.

> and if that's not the case, the threadID is also the first
> position in the thread:
> 
> $threadID[...]
> 
> right? so maybe its not such a big issue :)

Yes, if thread strees are all you are dealing with. In that case,
where do we store the thread strees?

Aside: For the API, I have been in database-thinking mode, and
trying to come up with something that can be manipulated easily
but also can be stored and retrieved well. It's awkward at times,
but since database operations will inevitably be the efficiency
bottleneck, I feel it's important to do that.

To get back to the question. My position is that it would be
better to have a memory copy of each forum's stree that gets
parsed down into individual thread strees as needed rather
than keeping all thread strees in memory, or, worse (in terms
of time efficiency) reading them from disk when needed.

Does that sound right?

> > What you were saying is clearly correct as well.
> 
> no need to walk on eggshells. bluntness is appreciated as always. I'd
> rather have folks yelling at me that i'm wrong than passively going along
> with every whim and fancy I have. Besides, it makes life more interesting
> :)

Glad to hear that, honestly. And bluntly, what you were saying
still is correct.

-- Ken

Re: [Iamb-dev-internal] API, development, etc

[Matt C. <su...@qx...>]

On Mon, 8 Jan 2001, Ken Rietz wrote:

> Check this out. We have to know how $stree is constructed in order
> to manipulate it correctly. This algorithm (I think) creates the
> $stree that Dan and I seemed to think best, except for putting
> in initial "[" and final "]" to reduce the number of cases in the
> adding items phase.

you had code for this (many moons ago), did you not?  

Also, let me get this on the record once and for all.  If this algorithm
could be substantially improved by being either in C via Inline.pm or in
XS, then do it.  Just keep an eye to porting while doing it.  Beyond that,
do what you think needs to be done to make this as fast as possible but
still maintainable.

> The first two would be simple: regexes would handle them well. The
> problem of finding matching "[" and "]" using regexes is hard, and
> inefficient to boot. However, it can be done by a straightforward
> character-by-character scan. That would be my first try.

cpan> install Text::Balanced

damian++

> Clearly, adding threads and articles is critical, so if this
> passes muster, then I'll work on the rest of the API as well
> as some of the code.

sounds good to me.

> One consideration is that no ID number--be it forum, thread, or
> article--can equal any other ID, or this whole process is doomed.

ok. forumIDs dont apply to strees as the strees are stored per thread not
per forum. so that eliminates one problem.  and threads are just articles,
yes?  the threadID is just the ID of the article at the head of the
Thread. and if that's not the case, the threadID is also the first
position in the thread:

$threadID[...]

right? so maybe its not such a big issue :)

> What you were saying is clearly correct as well.

no need to walk on eggshells. bluntness is appreciated as always. I'd
rather have folks yelling at me that i'm wrong than passively going along
with every whim and fancy I have. Besides, it makes life more interesting
:)

------
Matt Cashner
Web Applications Developer
Creative Group (http://www.cre8tivegroup.com)
su...@ea... | Codito, ergo sum

Re: [Iamb-dev-internal] API, development, etc

[Ken R. <kr...@qx...>]

Matt Cashner wrote:
> 
> On Sun, 7 Jan 2001, Ken Rietz wrote:
> 
> > Welcome back. I was getting concerned when you hadn't shown up
> > for a while.
> 
> my significant other was in town for the week. i will not see her until
> june, as she is in spain with a program through UK.  so you can understand
> where and why my attention has been the last week.

Makes it all clear.

> > Coming right up. Possibly with some code. First step, though, is
> > to define the characteristics of the $stree. I think I have most
> > of that already.
> 
> anything else you need?

Check this out. We have to know how $stree is constructed in order
to manipulate it correctly. This algorithm (I think) creates the
$stree that Dan and I seemed to think best, except for putting
in initial "[" and final "]" to reduce the number of cases in the
adding items phase.


Algorithm for formation of $stree
+++++++++++++++++++++++++++++++++
At start of forum:
$stree = '[' . "$forumID" . ']'


To add a thread or article:

1. Locate the $threadID (if responding directly to a thread) or
$articleID (if responding to an article in a thread) or $forumID (if
adding a new thread) in $stree

2. If the next character is a ',' or ']' (indicating no other articles
or threads at this level), insert, before that character,
'[' . "$articleID" . ']'
or
'[' . $threadID . ']'
as appropriate

3. If the next character is a '[' (indicating other articles or threads
at that level), locate the matching ']' and insert, after the ']',
',' . "$articleID"
or
',' . "$threadID"
as appropriate

------------

The first two would be simple: regexes would handle them well. The
problem of finding matching "[" and "]" using regexes is hard, and
inefficient to boot. However, it can be done by a straightforward
character-by-character scan. That would be my first try.

Clearly, adding threads and articles is critical, so if this
passes muster, then I'll work on the rest of the API as well
as some of the code.

One consideration is that no ID number--be it forum, thread, or
article--can equal any other ID, or this whole process is doomed.
The easiest way I know of to guarantee that is to store all of
these in one single database, and use the autoincrement key for
the ID of each. It was this I was trying (very unsuccessfully)
to argue before, namely that articles, threads and forums are
"the same". Profound apologies for not making this clear. :-(
What you were saying is clearly correct as well.

-- Ken

Re: [Iamb-dev-internal] API, development, etc

[Matt C. <su...@qx...>]

On Sun, 7 Jan 2001, Ken Rietz wrote:

> Welcome back. I was getting concerned when you hadn't shown up
> for a while.

my significant other was in town for the week. i will not see her until
june, as she is in spain with a program through UK.  so you can understand
where and why my attention has been the last week. 

> Coming right up. Possibly with some code. First step, though, is
> to define the characteristics of the $stree. I think I have most
> of that already.

anything else you need?

> Has the list of tasks been posted to sourceforge?

will take care of that tomorrow morning. 

------
Matt Cashner
Web Applications Developer
Creative Group (http://www.cre8tivegroup.com)
su...@ea... | Codito, ergo sum

Re: [Iamb-dev-internal] API, development, etc

[Matt C. <su...@qx...>]

On Sun, 7 Jan 2001, Rich Bowen wrote:

> Thus, we need a variety of things, which can be summarized as a new OO
> underpinnings for the whole thing. A user class, thread class, article
> class, and whatever else seems necessary. I don't expect that I'll ahve

well yeah. this was in my long range vision. i want to make sure this data
structure (stree's) will actually work before getting too far. but we will
definietly be rewriting the whole thing, with the major goal of doing this
right from step one with an eye toward scalability, speed, and
workability.

feature requests?

------
Matt Cashner
Web Applications Developer
Creative Group (http://www.cre8tivegroup.com)
su...@ea... | Codito, ergo sum

Re: [Iamb-dev-internal] API, development, etc

[Ken R. <kr...@qx...>]

Matt Cashner wrote:
> 

Welcome back. I was getting concerned when you hadn't shown up
for a while.

> Immediate Needs:
>     - Low-level API for stringified tree module [ken]

Coming right up. Possibly with some code. First step, though, is
to define the characteristics of the $stree. I think I have most
of that already.

Has the list of tasks been posted to sourceforge?

-- Ken

Re: [Iamb-dev-internal] API, development, etc

[Rich B. <rb...@rc...>]

> Can anyone think of anything else right off the bat?

Well, yeah. I've sort of determined that I have reached an impasse in
the "stable" tree, and it's time to start working on the other tree.
That is, if that's the sort of thing that the internals tree is for.

Thus, we need a variety of things, which can be summarized as a new OO
underpinnings for the whole thing. A user class, thread class, article
class, and whatever else seems necessary. I don't expect that I'll ahve
time to do this myself any time soon, but it's on the list.

It seems that all the other things that I want to do will break things
all over the place, and it's becoming harder to hack in the new features
that I want.

Does this seem reasonable?

-- 
Rich Bowen --  Director of Web Application Development
http://www.cre8tivegroup.com/  --  ri...@cr...
Have trouble remembering things?   http://www.idforgetmyhead.com/

[Iamb-dev-internal] API, development, etc

[Matt C. <su...@qx...>]

Immediate Needs:
    - Low-level API for stringified tree module [ken]

    - a better name for "stringified tree" (hereafter referred to as
      Stree's)

    - The actual Stree module (?)

    - IAMBd - a daemon that accepts and handles atomic changes to the
Stree's [matt]
	> i have a basic framework for such a daemon in POE thanks to some
other projects. All i need is a way to process the data. I will eventually
need a command set / query language for it but we'll cross that bridge
when we come to it.

   - I'd like to see some UI discussion. Tim's our resident expert on such
things. Is there a better UI we could be using? what are the
ramifications? etc.


Can anyone think of anything else right off the bat?

------
Matt Cashner
Web Applications Developer
Creative Group (http://www.cre8tivegroup.com)
su...@ea... | Codito, ergo sum