themis-dev — Development list.

[Themis-dev] 12/5/2008 Status

[Raymond C. R. <ra...@ba...>]

Hi folks,
 Earlier this week, I managed to put together a tiny Intel Atom 330 
based computer which can boot and run Haiku. Aside from needing to use 
the VESA video driver, and a quirk that keeps popping up when typing 
(Haiku seems to think I'm occasionally holding a key down long after 
I've released it), it seems beyond stable enough to do development on.

 In fact, if you belong to the themis-devcvs list, you probably have 
noticed the commits I've made to the Subversion repository over the last 
24 hours. Aside from the makefile, which I haven't committed yet, Themis 
and add-ons should now compile under Haiku. That's not to say everything 
is working like it was under BeOS, but it's a step in the right 
direction. I haven't committed the makefile yet because I had to remove 
the support for automatically generated dependencies: everytime I ran 
make, I would get an endless loop of updating the dependencies, and it 
wouldn't move on to building the object files. So, I went through and 
commented out all lines referencing the dependency files and generation 
thereof. I did make some Haiku specific additions to it as well, but 
they were relatively minor. (Mainly just making sure that it linked the 
framework against libbind.so, libsocket.so, libssl.so, and libcrypto.so. 
Yes, I'm going to be moving back over to OpenSSL.)

 I also had to check-out a copy of the repository via cvs to get a 
pristine copy of the themis.rsrc resource file to replace the copy that 
was in the Subversion repository. I'm guessing that when I converted the 
repository to svn years ago from cvs, that the resource file some how 
got corrupted in the process, and as a result was preventing xres from 
adding the resource file to the compiled Themis binary.
 
 The main problem that I've had outside of compiling is that the add-ons 
are not loading. I haven't investigated the reason they aren't loading 
yet, but all of them have compiled successfully, and are located where 
they are supposed to be located. Hopefully, I'll have the time this 
weekend to investigate this, and possibly even fix the problem(s).

 I intend to build a library out of the V8 Javascript engine that was 
developed as part of Google's Chrome web browser. I have checked out a 
copy of that repository and tried compiling it with scons (the build 
software they're using), but haven't gotten it to do much that way, 
probably because Haiku isn't on their supported list, so it wasn't taken 
into account in the SControl file. So, I've started looking for a way to 
circumvent this by building a makefile, but that's so far resulted in an 
error dealing with a static cast from one type to another using STL. I'm 
not very knowledgeable about STL, so this has halted my initial progress 
in that direction. I don't know if it's a genuine error in the code, or 
if it's just that gcc 2.95.x which is in use on BeOS and Haiku is too 
old to handle it. If you happen to know and/or are more knowledgeable 
about STL than I am, I'd appreciate a bit of help there.

 
That about covers things for the moment, but please let me know if you 
have any thoughts, comments, or ideas.

Raymond

Re: [Themis-dev] Themis re-organization

[Raymond C. R. <z3r...@bb...>]

Hi Kevin,
This is just a preliminary response, mainly to let you know that I'm
not just ignoring you. I'm cc'ing the mailing list so that others can
see my response as well. I want to apologize to you and the others for
the lack of response these last few months; part of the problem has been
a time issue (and the months did quickly slip by), part of the problem
has been a deluge of other emails and problems that I've needed to
address, and a good part has been just plain old laziness.

I would prefer  any general chit chat between multiple developers in a
meeting/conference form to take place on irc for several reasons: first,
it's a very simple and easily accessible protocol and system available
for free use across many operating systems and hardware platforms;
second, it allows active, moderated or unmoderated communication from
all interested parties; and third it is easily logged by all
participants. If there's another medium than can meet these criteria,
then I'm all for it.

I'll follow this up with a more detailed analysis of the portions of
Themis that I've personally written or worked on, but here's a quick
assessment from memory on the major components:

    *   The Framework Application
          In general pretty stable and offers a flexible platform for
loading and managing add-ons, and allowing "anonymous" interaction
between add-ons, meaning that the add-ons don't need to have any
specific intimate knowledge of one another to communicate and work with
each other. The add-on system needs to be expanded and adapted to handle
Netscape/Mozilla/Firefox add-ons but is pretty capable as is. The user
interface also needs to be tweaked, especially when it comes to the
application settings and the about box. The TCP communications could
also use a bit of debugging. While SSL support is currently working via
a non-OpenSSL toolkit, I've preferred OpenSSL overall for quite some
time. I don't believe that it (OpenSSL) compiles on BeOS or Haiku at
this time.

    *   The Cache System
          The cache system is extremely functional, though it could
stand a rewrite to be a bit more streamlined and for code
beautification. There is a possible flaw in that it creates an index on
all drives BFS for the cache file types, which allows Themis to query
for a cache file for any given URL regardless of which drive you might
have booted from. Some people might see that as a flaw, some might see
it as a feature. Personally I fall into the latter category, but
everyone has his or her own tastes.

    *   The HTTP Protocol and Cookies
          Cookies are working per the original RFC, and overall are very
stable. While they can be easily edited on disk, it would be a good idea
to add an in-application cookie editor and viewing utility. The HTTP
protocol support needs to be rewritten, and I would kill for BeOS/Haiku
to have PCRE built in to make processing HTTP headers easier, but
overall HTTP support is pretty stable and is mostly 1.1 compatible. It
is currently missing POST/PUT support (though that should be fairly
simple to add), and digest authentication, but it does feature basic
authentication.

    *   The Image Handler
          I was rewriting that last year in August, but lost all the
changes I made before I could commit them. The old one basically just
told the renderer add-on that any image it wanted was available, even if
it wasn't. It was just a stub more or less
because of some crashing issues I hadn't resolved years ago.

      *   The Makefile
          While it works, it is extremely ugly. I would love to move to
Jam if I could figure out how to use it. But I don't have even a weak
grip on how that system works.

    *   Javascript
        The SpiderMonkey javascript library was compiled and added to
the repository long ago, and is in desperate need of updating and better
understanding. While I got it to run the demo code that the Mozilla
project posted at the time, I never really figured out how to get it to
run a real Javascript script, much less really integrated into the rest
of the project.

I'll try to be a bit more complete at a later time (probably this week).

Raymond
ar...@co... wrote:
> Hey again,
>
> I have talked a little bit with Mark about Themis....
>
> I want to offer a suggestion and see if you might be interested.
>
> I would like to begin bringing in new developers to help out with Themis, however, I can't currently do that for several reasons:
>
> 1) I don't know what aspects of Themis are in good shape and which need to be re-done.
> 2) I don't now what parts are already called for (which ones you and Mark want to personally work on which ones are ok for other to come in an help with)
> 3) There is not much activity so that newcomers can ask questions when they come in.
>
>
> So...
> I wonder if you might consider setting up some kind of meeting in the near future to help get things ready for new developers to come in?
>
> I have attached a sample of what I kind of had in mind... though it doesn't have to be done that way at all if you don't want.
>
>
> If you think this would be a good route, then do you know of any way a discussion could be had that wouldn't involve IRC?  I suspect you probably want to use IRC... but was just wondering if any of the others could be used.
>
>
> Thanks,
> Kevin
>   

[Themis-dev] from Kevin throug Fredrik

[Fredrik <fr...@mo...>]

Hey, I wonder if you might be willing to copy the email below and repost
it on the
mailing list?  The mailing list is rejecting my messages for some reason.
Kevin
--------------------------------------

To Fredrik (who has been doing some work on the UI), to Michael who wants to
work on the UI soon??, and to the rest of the Themis devs who have a lot
of work
put into the project....

I contacted Remi Grumeau about some help with the UI.

In case you were not aware, Remi did some mockups for a possible UI for a
Haiku
browser project called Net++ that was supposed to just focus on a better UI,
while using Gecko for the core stuff.  According to a post he made, the
project
never really got started apart from his mockups.

You can see his Net++ mockups here:
http://beosfrance.com/muckups/Net++/
you can also check out his flash demo in the flash subfolder
He has also done mockups for other stuff:
http://beosfrance.com/muckups/

Anyways, I noticed that there was a little uncertainty about what to do
with the
UI.  What design/layout to use... do we just end up copying code from
NetPositive.  Although Remi cannot code C++, perhaps you all might be
interested
in working with him on the UI design?  I'm not saying his ideas are the
best (they
may be too complicated?),
but I think he
might be able to help.
If Fredrik and Michael and any others working on the UI feel like working
with
Remi... maybe you all can come up with a design that isn't just a copy of
other
browsers (like Firefox) and maybe it means you wouldn't have to worry as much
about copying code from NetPositive.

If Remi can ideed help, maybe it would take a little pressure off of the
coding
of the UI...
* perhaps Remi can add additional creativity to the UI design...
* maybe it would mean no longer copying Firefox's design/layout ... instead
maybe we can think about new things and a more Haiku-like design (this is a
Haiku native app for a completely new browser system after all :) )

Either, I just wanted to run things by you.  Although Remi is not a coder, I
wonder if you would like him to come in and help out with the UI design
(meaning
work together with him on the design)?

As a side note, I kinda realize that finalizing a UI may or may not be kinda
premature at this point... as
the major issues are getting coding
done on the core parsing and rendering and other aspects of the browser. 
Then
again, maybe this would just mean having a great UI once the underlying
stuff is
nearly ready (which is nice :) ), so what do you all think?

I'll probably tell him I posted this message and let him decide if he
wants to
go ahead and post here or wait for a response. :)

Fredriks responce :) :
I'm more of an C++/C# coder than an UI designer so if there was a UI plan
that would be good to have am follow. About copy code it's only function
and not how it's looks :).
So yes but I think that all of us need to work with him :).

By the way could we have a branch or some thing I have some stuff that are
not in svn so I would be glad if I had access to commit. In a week I have
abut (2*30)*5  = 5 hours to work on themis as I have a small Dell x300 and
do moste of this work to and from work.

Re: [Themis-dev] Status of Themis

[Mark H. <ma...@fi...>]

>othern than these people are there other that at'nt on the about list?
>
>Who's involved:
>Mark (planning to later on)
>Raymond (planning to later on)
>Michael (planning to?)
>Fredrik (working now - UI)
>Maxwell (working now - CSS)
>
>What did these people do?
>Linus Almstrom

General support, tips and suggestions. Maybe some more things. Raymond ?

>Olivier Milla

Worked on the renderer. The renderer module is his.

>I want to update the about box with som stuff :)
>today we have in the about box:
>Themis framework
>SSL
>
>missing are:
>XML/HTML Parser    - Mark
>CSS                - Axel/Maxwell?
>JavaScript         - ?
>DOM                - ?

The Javascript engine is spidermonkey from Mozilla.
The DOM is written by me, but it falls under the framework.
Perhaps we have to pull some information directly from the modules. I
don't know what information we are currently storing in there.

>Have I missed an area /or Person or have one to many?
>
>About coding style I think that we should follow Haiku?

Yeah, I think they follow OpenTracker coding style, right ?
We don't have a  coding style at the moment, but I think those look good.
Raymond what do you think ?

Btw, I browsed through the diff you sent to the mailinglist a bit and it
looks ok to me.
If you want to have write access to the repository and Raymond is ok with
it, he can give you access.

Mark

-- 
Spangalese for beginnners:

Re: [Themis-dev] Status of Themis

[Fredrik <fr...@mo...>]

othern than these people are there other that at'nt on the about list?

Who's involved:
Mark (planning to later on)
Raymond (planning to later on)
Michael (planning to?)
Fredrik (working now - UI)
Maxwell (working now - CSS)

What did these people do?
Linus Almstrom
Olivier Milla

I want to update the about box with som stuff :)
today we have in the about box:
Themis framework
SSL

missing are:
XML/HTML Parser    - Mark
CSS                - Axel/Maxwell?
JavaScript         - ?
DOM                - ?

Have I missed an area /or Person or have one to many?

About coding style I think that we should follow Haiku?

//Fredrik

>>Well, I can give some of the info I know...
>>
>>Who's involved:
>>Mark (planning to later on)
>>Raymond (planning to later on)
>>Michael (planning to?)
>>Fredrik (working now - UI)
>>Maxwell (working now - CSS)
>>
>>Who's who:
>>Mark - longtime Themis developer
>>Raymond - started the Themis project?
>
> That's correct. Raymond started it.
>
>>Michael - longtime Themis developer -- UI and other areas
>>Fredrik - ?
>>Maxwell - new to the project; has written a CSS parser on his own :)
>>
>>Past contributers:
>>Alex - contributed the CSS parser we have now
>>
>>Other people:
>>Cyril - working on a separate project called UZI - I asked him to make
> any notes about problems he noticed in Themis (so we can consider all
> views) since he evaluated Themis before working on his own browser
>>
>>Status of components:
>>
>>HTML parser -- Marks wants to do a rewrite -- do you need help??
>
> No, I just need quite a bit of time. I'll see if I can merge my changes
> into what is currently available, so it is at least off my computer. I do
> want the parser to keep working if I commit what I currently have, even
> though it is not very good at the moment.
>
>>EMCAScript + DOM -- there's rough Spidermonkey support?  I was hoping to
> find someone to do some work on
>> EMCAScript -- perhaps even using the SEE EMCAScript Interpreter or the
> LLVM Compiler.
>
> I don't think it actually does anything, so as far as I know all options
> are still on the table here.
>
>>UI -- Fredrik -- need help?
>>
>>SVG -- nothing now; no plans this early on
>>MathML -- nothing; no plans this early on
>
> Nope.
>
>>XML?
>
> Basically supported with the html parser as the parser is actually an sgml
> parser.
>
>>XSLT?
>>XPATH?
>>XForms?
>>etc....???  what other areas can we add that will be needed in the
> future?
>
> X3D, VRML, SMIL, Java, Flash, etc. ;)
>
>>
>>Additional questions:
>>Both Mark and Raymond have indicated that many areas needed rewrites or
> that they wanted to do rewrites on
>> certain areas.  I think this information would be very, very useful to
> know about.  Is it possible to get a
>> complete list of every one of those items? :)
>>  which parts have a good basis and just need to be polished little,
> which areas need a rewrite, whether you
>> want to do the rewrite yourself or if someone else needs to be brought
> in, and so on....
>>  Perhaps this info would be helpful?
>
> Simple overview (from my part of the code):
> html parser: rewrite in progress.
> dom: works fine, needs to implement more parts of the spec, but no real
> rush.
> css-dom: basics are there, but needs serious looking at. All options are
> on the table, as far as I am concerned.
>
> Mark
>
> --
> Spangalese for beginnners:
> `Lunue d'nent.'
> `Your axe is swift Stewardess.'
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Themis-dev mailing list
> The...@li...
> https://lists.sourceforge.net/lists/listinfo/themis-dev
>


-- 
MVH
Fredrik Modéen

Re: [Themis-dev] Status of Themis

[Mark H. <ma...@fi...>]

>Well, I can give some of the info I know...
>
>Who's involved:
>Mark (planning to later on)
>Raymond (planning to later on)
>Michael (planning to?)
>Fredrik (working now - UI)
>Maxwell (working now - CSS)
>
>Who's who:
>Mark - longtime Themis developer
>Raymond - started the Themis project?

That's correct. Raymond started it.

>Michael - longtime Themis developer -- UI and other areas
>Fredrik - ?
>Maxwell - new to the project; has written a CSS parser on his own :)
>
>Past contributers:
>Alex - contributed the CSS parser we have now
>
>Other people:
>Cyril - working on a separate project called UZI - I asked him to make
any notes about problems he noticed in Themis (so we can consider all
views) since he evaluated Themis before working on his own browser
>
>Status of components:
>
>HTML parser -- Marks wants to do a rewrite -- do you need help??

No, I just need quite a bit of time. I'll see if I can merge my changes
into what is currently available, so it is at least off my computer. I do
want the parser to keep working if I commit what I currently have, even
though it is not very good at the moment.

>EMCAScript + DOM -- there's rough Spidermonkey support?  I was hoping to
find someone to do some work on
> EMCAScript -- perhaps even using the SEE EMCAScript Interpreter or the
LLVM Compiler.

I don't think it actually does anything, so as far as I know all options
are still on the table here.

>UI -- Fredrik -- need help?
>
>SVG -- nothing now; no plans this early on
>MathML -- nothing; no plans this early on

Nope.

>XML?

Basically supported with the html parser as the parser is actually an sgml
parser.

>XSLT?
>XPATH?
>XForms?
>etc....???  what other areas can we add that will be needed in the
future?

X3D, VRML, SMIL, Java, Flash, etc. ;)

>
>Additional questions:
>Both Mark and Raymond have indicated that many areas needed rewrites or
that they wanted to do rewrites on
> certain areas.  I think this information would be very, very useful to
know about.  Is it possible to get a
> complete list of every one of those items? :)
>  which parts have a good basis and just need to be polished little,
which areas need a rewrite, whether you
> want to do the rewrite yourself or if someone else needs to be brought
in, and so on....
>  Perhaps this info would be helpful?

Simple overview (from my part of the code):
html parser: rewrite in progress.
dom: works fine, needs to implement more parts of the spec, but no real
rush.
css-dom: basics are there, but needs serious looking at. All options are
on the table, as far as I am concerned.

Mark

-- 
Spangalese for beginnners:
`Lunue d'nent.'
`Your axe is swift Stewardess.'

Re: [Themis-dev] Status of Themis

[Fredrik <fr...@mo...>]

New to Themis .Have worked on Quake1 port that started by aiXplosive some
changes in Quake3, have some project by my self but never finished, card
game called Spiderette, genealogy program was thinking of making a clone
of the game risk but didn't hade the time. Tried BeOS when R3 was new for
the first time but didn't know c++ then. Working as a programmer in C#
today (worked as a data technician when R3 was new).

Oh nearly forgot I have made some updates of Transmission but my last
changes haven’t been integrated and Transmission after 0.82 are
broken due to libevent not working in BeOS/Zeta/Haiku.

>> I will start working on the html parser again to finally finish the
>> rewrite, but of course a lot of other work needs to be done.
>> Who else wants to work on Themis and on which part do you want to work ?
>> Raymond, do you have the time ?
>> I think someone volunteered for the GUI ? ;)
>> I don't think we want to have any timelines about when something should
>> be
>> finished, although more progress than the last two years should be the
>> minimum. :)
>
> Well, I can give some of the info I know...
>
> Who's involved:
> Mark (planning to later on)
> Raymond (planning to later on)
> Michael (planning to?)
> Fredrik (working now - UI)
> Maxwell (working now - CSS)
>
> Who's who:
> Mark - longtime Themis developer
> Raymond - started the Themis project?
> Michael - longtime Themis developer -- UI and other areas
> Fredrik - ?

> Maxwell - new to the project; has written a CSS parser on his own :)
>
> Past contributers:
> Alex - contributed the CSS parser we have now
>
> Other people:
> Cyril - working on a separate project called UZI - I asked him to make any
> notes about problems he noticed in Themis (so we can consider all views)
> since he evaluated Themis before working on his own browser
>
> Status of components:
>
> HTML parser -- Marks wants to do a rewrite -- do you need help??
> CSS parser -- We have a handwritten CSS parser from Alex, but for the
> future, a decision must be made; Maxwell is currently evaluating parsers
> to see what the best options might be before making any suggestions in the
> mailing list
> EMCAScript + DOM -- there's rough Spidermonkey support?  I was hoping to
> find someone to do some work on EMCAScript -- perhaps even using the SEE
> EMCAScript Interpreter or the LLVM Compiler.
> UI -- Fredrik -- need help?
>
> SVG -- nothing now; no plans this early on
> MathML -- nothing; no plans this early on
>
> XML?
> XSLT?
> XPATH?
> XForms?
> etc....???  what other areas can we add that will be needed in the future?
>
> Additional questions:
> Both Mark and Raymond have indicated that many areas needed rewrites or
> that they wanted to do rewrites on certain areas.  I think this
> information would be very, very useful to know about.  Is it possible to
> get a complete list of every one of those items? :)  which parts have a
> good basis and just need to be polished little, which areas need a
> rewrite, whether you want to do the rewrite yourself or if someone else
> needs to be brought in, and so on....  Perhaps this info would be helpful?
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Themis-dev mailing list
> The...@li...
> https://lists.sourceforge.net/lists/listinfo/themis-dev
>


-- 
MVH
Fredrik Modéen

Re: [Themis-dev] Status of Themis

[<Ar...@co...>]

Correction... it's Axel... as in Axel Dorfler, not Alex, who wrote the CSS parser.

I sent an email to Axel to see if he still has interest in stuff like this, but with his work on Haiku it's probably not likely. :)
Also, I contacted Remi, but if someone knows a better way to contact him, feel free.  He did some UI stuff here:
http://www.beosfrance.com/muckups/Net++/
not coding, but more design ideas...  might be useful to have some more ideas to consider

 -------------- Original message ----------------------
From: Ar...@co...
> > I will start working on the html parser again to finally finish the
> > rewrite, but of course a lot of other work needs to be done.
> > Who else wants to work on Themis and on which part do you want to work ?
> > Raymond, do you have the time ?
> > I think someone volunteered for the GUI ? ;)
> > I don't think we want to have any timelines about when something should be
> > finished, although more progress than the last two years should be the
> > minimum. :)
> 
> Well, I can give some of the info I know...
> 
> Who's involved:
> Mark (planning to later on)
> Raymond (planning to later on)
> Michael (planning to?)
> Fredrik (working now - UI)
> Maxwell (working now - CSS)
> 
> Who's who:
> Mark - longtime Themis developer
> Raymond - started the Themis project?
> Michael - longtime Themis developer -- UI and other areas
> Fredrik - ?
> Maxwell - new to the project; has written a CSS parser on his own :)
> 
> Past contributers:
> Alex - contributed the CSS parser we have now
> 
> Other people:
> Cyril - working on a separate project called UZI - I asked him to make any notes 
> about problems he noticed in Themis (so we can consider all views) since he 
> evaluated Themis before working on his own browser
> 
> Status of components:
> 
> HTML parser -- Marks wants to do a rewrite -- do you need help??
> CSS parser -- We have a handwritten CSS parser from Alex, but for the future, a 
> decision must be made; Maxwell is currently evaluating parsers to see what the 
> best options might be before making any suggestions in the mailing list
> EMCAScript + DOM -- there's rough Spidermonkey support?  I was hoping to find 
> someone to do some work on EMCAScript -- perhaps even using the SEE EMCAScript 
> Interpreter or the LLVM Compiler.
> UI -- Fredrik -- need help?
> 
> SVG -- nothing now; no plans this early on
> MathML -- nothing; no plans this early on
> 
> XML?
> XSLT?
> XPATH?
> XForms?
> etc....???  what other areas can we add that will be needed in the future?
> 
> Additional questions:
> Both Mark and Raymond have indicated that many areas needed rewrites or that 
> they wanted to do rewrites on certain areas.  I think this information would be 
> very, very useful to know about.  Is it possible to get a complete list of every 
> one of those items? :)  which parts have a good basis and just need to be 
> polished little, which areas need a rewrite, whether you want to do the rewrite 
> yourself or if someone else needs to be brought in, and so on....  Perhaps this 
> info would be helpful?
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Themis-dev mailing list
> The...@li...
> https://lists.sourceforge.net/lists/listinfo/themis-dev

Re: [Themis-dev] Status of Themis

[<Ar...@co...>]

> I will start working on the html parser again to finally finish the
> rewrite, but of course a lot of other work needs to be done.
> Who else wants to work on Themis and on which part do you want to work ?
> Raymond, do you have the time ?
> I think someone volunteered for the GUI ? ;)
> I don't think we want to have any timelines about when something should be
> finished, although more progress than the last two years should be the
> minimum. :)

Well, I can give some of the info I know...

Who's involved:
Mark (planning to later on)
Raymond (planning to later on)
Michael (planning to?)
Fredrik (working now - UI)
Maxwell (working now - CSS)

Who's who:
Mark - longtime Themis developer
Raymond - started the Themis project?
Michael - longtime Themis developer -- UI and other areas
Fredrik - ?
Maxwell - new to the project; has written a CSS parser on his own :)

Past contributers:
Alex - contributed the CSS parser we have now

Other people:
Cyril - working on a separate project called UZI - I asked him to make any notes about problems he noticed in Themis (so we can consider all views) since he evaluated Themis before working on his own browser

Status of components:

HTML parser -- Marks wants to do a rewrite -- do you need help??
CSS parser -- We have a handwritten CSS parser from Alex, but for the future, a decision must be made; Maxwell is currently evaluating parsers to see what the best options might be before making any suggestions in the mailing list
EMCAScript + DOM -- there's rough Spidermonkey support?  I was hoping to find someone to do some work on EMCAScript -- perhaps even using the SEE EMCAScript Interpreter or the LLVM Compiler.
UI -- Fredrik -- need help?

SVG -- nothing now; no plans this early on
MathML -- nothing; no plans this early on

XML?
XSLT?
XPATH?
XForms?
etc....???  what other areas can we add that will be needed in the future?

Additional questions:
Both Mark and Raymond have indicated that many areas needed rewrites or that they wanted to do rewrites on certain areas.  I think this information would be very, very useful to know about.  Is it possible to get a complete list of every one of those items? :)  which parts have a good basis and just need to be polished little, which areas need a rewrite, whether you want to do the rewrite yourself or if someone else needs to be brought in, and so on....  Perhaps this info would be helpful?

Re: [Themis-dev] Status of Themis

[Fredrik <fr...@mo...>]

> Hello everybody,
>
> there has been a lot of e-mails on this list lately about what to do with
> Themis.
> This is almost entirely due to Kevin's interest in the project.
> Thanks for that, Kevin!
>
> I have said in a previous mail that working on Themis again was pretty
> high on my wish list.
> I thought about it for quite a while and while I realize it will take a
> lot of time and hard work, I want to try to pick it up again. That doesn't
> mean I will start contributing code again tomorrow, but that does mean
> that I will do my best to get Themis to a more usable state instead of
> just watching what happens in the BeOS world.
>
> I will start working on the html parser again to finally finish the
> rewrite, but of course a lot of other work needs to be done.
> Who else wants to work on Themis and on which part do you want to work ?
> Raymond, do you have the time ?
> I think someone volunteered for the GUI ? ;)
like me!? :)
> I don't think we want to have any timelines about when something should be
> finished, although more progress than the last two years should be the
> minimum. :)
>
> Mark
>
> --
> Spangalese for beginnners:
> `Lunue d'nent.'
> `Your axe is swift Stewardess.'
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Themis-dev mailing list
> The...@li...
> https://lists.sourceforge.net/lists/listinfo/themis-dev
>


-- 
MVH
Fredrik Modéen

[Themis-dev] Status of Themis

[Mark H. <ma...@fi...>]

Hello everybody,

there has been a lot of e-mails on this list lately about what to do with
Themis.
This is almost entirely due to Kevin's interest in the project.
Thanks for that, Kevin!

I have said in a previous mail that working on Themis again was pretty
high on my wish list.
I thought about it for quite a while and while I realize it will take a
lot of time and hard work, I want to try to pick it up again. That doesn't
mean I will start contributing code again tomorrow, but that does mean
that I will do my best to get Themis to a more usable state instead of
just watching what happens in the BeOS world.

I will start working on the html parser again to finally finish the
rewrite, but of course a lot of other work needs to be done.
Who else wants to work on Themis and on which part do you want to work ?
Raymond, do you have the time ?
I think someone volunteered for the GUI ? ;)
I don't think we want to have any timelines about when something should be
finished, although more progress than the last two years should be the
minimum. :)

Mark

-- 
Spangalese for beginnners:
`Lunue d'nent.'
`Your axe is swift Stewardess.'

Re: [Themis-dev] CSS

[<Ar...@co...>]

>   It's funny you point me to Elkhound. I've made a project few years 
> ago, where I needed an C++ parser, and, by that time contacted
> Elkhound/Elsa author. The project is on hold (basically because one of 
> the resource it relied upon is down), but the idea was to create
> an C++ IDE running on the Pocket PC platform, able to compile C++ code 
> directly (without requiring any host PC like it's the case now).
> The project is working ( http://lewort.free.fr ), but the compiler is no 
> more available, so it's useless (unless some open-source genius
> succeed in creating a C++ compiler other that GCC)
Well you are in for a nice surprise.  There is work being done on a GCC alternative -- it's called LLVM.  I've spent some time working on their website.
A few things about LLVM:
*) some people from Apple contribute to it; in fact, the person who started it now works for Apple
*) License: BSD
*) faster parsing than GCC
*) less memory usage than GCC
*) better diagnostic/error messages than GCC
*) easier for people to work on the compiler itself (as opposed to GCC, which is supposedly hard to work on)
*) library based design that will allow for specialized debugging tools, IDE tools, etc... -- not possible with GCC

As it stands right now, LLVM currently relies on GCC for the front-end.  The website I had done some work on is for a subprojet of LLVM to create a C, Objective C, and C++ front-end.

Personally, I really love to see Haiku switch away from GCC and to LLVM in the future (in fact, it's possible to even do that now), but anyways.... :)

Re: [Themis-dev] CSS

[Cyril R. <sta...@la...>]

Ar...@co... a écrit :
> Is Elkhound of any use to you?
> http://www.bestechvideos.com/2006/12/04/elkhound-elsa-and-cqual-open-source-static-analysis-for-c
> I know it's for compilers, but there might be some overlap and/or concepts that might be useful??
>   
Sorry for replying to everyone to what should have been a private mail.
If an admin want to delete the previous post, please do.

Regards,
Cyril

Re: [Themis-dev] CSS

[Cyril R. <sta...@la...>]

Ar...@co... a écrit :
> Is Elkhound of any use to you?
> http://www.bestechvideos.com/2006/12/04/elkhound-elsa-and-cqual-open-source-static-analysis-for-c
> I know it's for compilers, but there might be some overlap and/or concepts that might be useful??
>   
Hi Kevin,

  It's funny you point me to Elkhound. I've made a project few years 
ago, where I needed an C++ parser, and, by that time contacted
Elkhound/Elsa author. The project is on hold (basically because one of 
the resource it relied upon is down), but the idea was to create
an C++ IDE running on the Pocket PC platform, able to compile C++ code 
directly (without requiring any host PC like it's the case now).
The project is working ( http://lewort.free.fr ), but the compiler is no 
more available, so it's useless (unless some open-source genius
succeed in creating a C++ compiler other that GCC)

BTW, I've computed a slideshow showing some concepts of UZI at  
http://uziproject.free.fr/concept.html
I'm finally trying to roll my own license, and let you know when I'm done.

Sincerely,
Cyril

Re: [Themis-dev] CSS

[<Ar...@co...>]

Is Elkhound of any use to you?
http://www.bestechvideos.com/2006/12/04/elkhound-elsa-and-cqual-open-source-static-analysis-for-c
I know it's for compilers, but there might be some overlap and/or concepts that might be useful??



 -------------- Original message ----------------------
From: Maxwell Collins <ma...@um...>
> Yes, I've been following the mailing list.
> 
> I'm currently doing a more in-depth analysis of the CSS grammar
> (currently trying to determine if the CSS3 grammar is LL(1)), so I can
> better judge the available parsers.  There is a number of really odd
> properties of the CSS grammar, and one of the most important properties
> of a CSS parser is how it handles these.  I'm going to completely
> analyze all of the oddities of the CSS 2.1 grammar (which I'm familiar
> with), then try to see if I can anticipate any similar issues that may
> have arisen with some of the new CSS3 stuff.
> 
> Once I've completed that, I'll send my thoughts on all of the available
> CSS Parsers/CSSOM packages (including some beyond the three you listed).
> 
> On Fri, 2007-09-14 at 06:32 +0000, Ar...@co... wrote:
> > Hey,
> > Are you still following the mailing list?
> > I posted something about a person working on a project called UZI....  Have 
> you looked at any of the stuff he has to see if it might be something worth 
> considering for Themis?
> > 
> > Anyways, what's your thoughts on CSS (if you've gotten a chance to look)?  
> Does the current CSS look like it's a good start, does it look like the 
> foundation is bad and that it would just be better to start with a new CSS 
> parser (like yours).  I know you were interested in the particular area of CSS, 
> but the reason I'm contacting you is to see how UZI mixes into things.  In 
> comparing the 3: Themis' CSS, your CSS, or UZI's CSS, which appears to have the 
> more solid foundation (regardless of whether it's even close to feature 
> complete)... and which is closer to feature complete?
> > Of course, what involvement UZI might have in the Themis project is uncertain 
> at this point, but I didn't want their to be a conflict between what you were 
> interested in and that project (since that project also include CSS stuff).
> > 
> > Kevin
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Themis-dev mailing list
> The...@li...
> https://lists.sourceforge.net/lists/listinfo/themis-dev

Re: [Themis-dev] Another code base

[Cyril R. <sta...@la...>]

Raymond C. Rodgers a écrit :
> Cyril Russo wrote:
>> Well, here are the papers I'm referring about:
>> http://www.cs.umass.edu/%7Eemery/pubs/berger-oopsla2002.pdf  (about 
>> using an allocator can speed up the allocation time by 40%)
>> And the older one:
>> http://www.cs.umass.edu/%7Eemery/pubs/berger-pldi2001.pdf (a basic 
>> example on heap layers)
>>
>> I have nothing about threading a parser. However, speaking about data 
>> flow, I would say that:
>>   For the rendering to be done, you need the entire DOM tree (just think 
>> of absolutely positioned or floating elements for one). You also need 
>> all the linked resources (whatever, CSS, IMG, etc...)
>>   For the DOM tree to be entire, you need HTML source to be entire.
>>   For the CSS resource, you also need HTML source to be entire (as, even 
>> if <style> tag is only allowed in <head>, it's very frequent to find 
>> them in the body, google is a nice example of that).
>>  
>> So yes, surely, you can start parsing an HTML stream, and start making a 
>> (partial) DOM tree from it, and still start rendering it, all at the 
>> same time.
>> However, if there is a table in the page, you'll break everything not 
>> waiting for table end (as table size is computable only when received 
>> entirely), and table based layout at clearly too frequent to be ignored.
>> For example, a <p> tag inside a <table> tag should be rendered before 
>> the table (it's illegal, but that's how Mozilla and IE does).
>> If there is some JS, then you'll also have to get the whole things 
>> before rendering as JS usually deals with DOM tree (so it has to be 
>> entire if you don't want your JS engine to fail)
>> Finally, rendering prematuraly usually gives the flashing effect you had 
>> on Firefox 1.0 when the CSS didn't loaded in time (the <entire> DOM tree 
>> rendered, then the CSS arrived, and the <entire> DOM tree rerendered, 
>> moving things around).
>>
>> Anyway, I don't get the interest of multithreading the parser & renderer 
>> (mainly for complexifying the parser, because of the multiple locks 
>> required to avoid data contention) if you are not going to render the 
>> final page until all steps are done.
>> On single core CPU, you'll never get any improvment (well, dataflow 
>> analysis), and on multicore CPU, the locking mechanism (and bad cache 
>> data) might simply break any gain you're expecting (as, at best, the 
>> rendering time will be : max(parser time (you need the whole document 
>> here), fetching time (way longer that anything else), CSS-DOM mapping, 
>> renderer) so it's likely = to fetching time)
>>
>>   
> I'm not an expert on anything, period. But it seems logical to me that 
> a document could be rendered more quickly with two threads than with a 
> single one if the operating system's scheduler is designed with 
> multithreading in mind, as BeOS was. I have seen performance 
> improvements in BeOS applications when adding a second thread 
> performing the same functionality, if for no other reason than the 
> BeOS scheduler sees it as another item that needs attention before it 
> begins cycling through the process list again. Besides the idea of 
> attempting to render more than one part of a document at a time with 
> multiple threads, there's also the more practical reality of adding 
> threads to the parser and renderer: multiple simultaneous documents. 
> When dealing with Firefox,  IE 7, or Opera 9, it is not uncommon to 
> open multiple documents simultaneously. If you have a parser that is 
> not multithreaded, you end up with a queue system where each document 
> must be processed sequentially, and then rendered sequentially.
>
> Take this example, you open a bookmark that opens 3 tabs 
> automatically. Tab 1 opens cnn.com, tab 2 opens slashdot.org, and tab 
> 3 opens an image stored on your hard drive. Without being 
> multithreaded, the browser might have to wait for cnn.com and 
> slashdot.org to be retrieved, parsed, and rendered before it could 
> load the image from your hard disk. Or, assuming that your retrieval 
> system is multithreaded, and renders whatever is completely retrieved 
> first, you still end up waiting for one tab to be completely done 
> before the next can be completed. This example obviously isn't 
> completely practical, as no one can read multiple tabs simultaneously, 
> but the point is still the same: being multithreaded allows multiple 
> documents to be parsed and rendered simultaneously.
>
> As for the messaging system and event driven application design, I 
> agree with Mark in that it serves our purposes better. In the same 
> example as above, the image in tab 3 has no need to go through a 
> SGML/HTML/XML parser. It's an image, and we know it's an image, why do 
> we need to send it there? In the design we used with Themis, and will 
> be implementing in the renderer and image handler, the image handler 
> will see that an image document has been retrieved and will process it 
> for the renderer, and in the process notify the renderer (and anyone 
> else that might be interested) that the image is ready to use. The 
> renderer can then use the image when it's ready for it, which could be 
> right away in the example above.
>
> In the cases of tabs 1 and 2 (cnn.com and slashdot.org respectively), 
> the images could be added to the rendered output as they become 
> available. The reason I state "as they become available" is simple: 
> when I browse the web, I'm usually much more interested in the text 
> content of a page than the images or other media unless that's what 
> I'm looking for. I'd much rather have CNN's content appear so I can 
> start looking for articles that I'm interested in while the images are 
> being loaded than having to wait for the whole page (images and all) 
> to be completely  retrieved first.
You're right.
You are describing an usage pattern, while I was describing a dataflow 
pattern.
To carry on your example, if you have multiple tab, then each tab in UZI 
have its own thread doing successively (parsing / mapping / rendering).
However, the fetcher is shared between threads.

Concerning the picture you're talking about, the URL handler code (in 
Network/URLHandler.hpp) will find by itself that the address (to the 
picture) is a local file, and then fetch it (load from disk) in one of 
the thread pool. Then the cache engine will then determine the file 
type, which will most probably result in JPG picture. As it's not HTML 
or XHTML, the HTML parser will not be called, nor will the DOM mapper, 
only the renderer will be called with a JPG document as source (and not 
a dom tree).

Finally, I haven't paid for the lock overhead in the parser/mapper/renderer.
There is no need to protect data in every tab, as no tab can deal with 
content from other tab.
I'm paying for locking overhead in the fetcher (so each call to the 
fetcher are serialized).

Surely, when only one tab is open or actively used (as it's the case for 
most of the time), the time to see a document in UZI requires that the 
parser finished parsing, the mapper finished mapping, and the renderer 
finished, well, rendering.

In your structure, you are correct about this, the renderer will start 
displaying data while the parser hasn't finished parsing, at the cost of 
data synchronization, and rerendering when everything is 
received/parsed. It might be more "reactive" to user POV, but at the 
same time very very more difficult for the dev POV.


>> That why I've implemented multithreaded fetching in UZI, so that the 
>> rendering time will be (deterministic), max(fetching time + rendering, 
>> parsing + CSS mapping + rendering), and the code still remains clear of 
>> locking primitives.
>>
>>   
> You more than likely will need to have locking of some sort in this 
> design, especially when dealing with Linux. Sending and receiving data 
> should always be done under lock conditions when done in a 
> multithreaded fashion to prevent corruption. Many of the networking 
> functions are not reentrant, and can easily cause data corruption as a 
> result.
I disagree on that point. All the networking functions are reentrant, 
provided you use a recent libc, as it's a POSIX requirement.
However, I agree on the fact that it's not because the function are 
reentrant that it's possible to call them without locking your data first.
The fetcher locks the data while it fetches them, and release it while 
it's "select"ing the socket. That way, it's possible to get some stats 
on the communication, and, in a later step, start parsing the received 
data while receiving them.
However, the finer the locking mechanism is, the longer you'll loose in 
system time.

In UZI, I've choosen the minimum possible locking.
I understand your choices through, but warn you about the difficulties 
you'll encounter (specially the about the parsing error leading to full 
rework of the DOM tree, hence current rendering, and so on).
Sometime it's easier to complexify a simple design than doing the opposite.

Cyril

Re: [Themis-dev] Another code base

[Raymond C. R. <z3r...@bb...>]

Cyril Russo wrote:
> Well, here are the papers I'm referring about:
> http://www.cs.umass.edu/%7Eemery/pubs/berger-oopsla2002.pdf  (about 
> using an allocator can speed up the allocation time by 40%)
> And the older one:
> http://www.cs.umass.edu/%7Eemery/pubs/berger-pldi2001.pdf (a basic 
> example on heap layers)
>
> I have nothing about threading a parser. However, speaking about data 
> flow, I would say that:
>   For the rendering to be done, you need the entire DOM tree (just think 
> of absolutely positioned or floating elements for one). You also need 
> all the linked resources (whatever, CSS, IMG, etc...)
>   For the DOM tree to be entire, you need HTML source to be entire.
>   For the CSS resource, you also need HTML source to be entire (as, even 
> if <style> tag is only allowed in <head>, it's very frequent to find 
> them in the body, google is a nice example of that).
>  
> So yes, surely, you can start parsing an HTML stream, and start making a 
> (partial) DOM tree from it, and still start rendering it, all at the 
> same time.
> However, if there is a table in the page, you'll break everything not 
> waiting for table end (as table size is computable only when received 
> entirely), and table based layout at clearly too frequent to be ignored.
> For example, a <p> tag inside a <table> tag should be rendered before 
> the table (it's illegal, but that's how Mozilla and IE does).
> If there is some JS, then you'll also have to get the whole things 
> before rendering as JS usually deals with DOM tree (so it has to be 
> entire if you don't want your JS engine to fail)
> Finally, rendering prematuraly usually gives the flashing effect you had 
> on Firefox 1.0 when the CSS didn't loaded in time (the <entire> DOM tree 
> rendered, then the CSS arrived, and the <entire> DOM tree rerendered, 
> moving things around).
>
> Anyway, I don't get the interest of multithreading the parser & renderer 
> (mainly for complexifying the parser, because of the multiple locks 
> required to avoid data contention) if you are not going to render the 
> final page until all steps are done.
> On single core CPU, you'll never get any improvment (well, dataflow 
> analysis), and on multicore CPU, the locking mechanism (and bad cache 
> data) might simply break any gain you're expecting (as, at best, the 
> rendering time will be : max(parser time (you need the whole document 
> here), fetching time (way longer that anything else), CSS-DOM mapping, 
> renderer) so it's likely = to fetching time)
>
>   
I'm not an expert on anything, period. But it seems logical to me that a 
document could be rendered more quickly with two threads than with a 
single one if the operating system's scheduler is designed with 
multithreading in mind, as BeOS was. I have seen performance 
improvements in BeOS applications when adding a second thread performing 
the same functionality, if for no other reason than the BeOS scheduler 
sees it as another item that needs attention before it begins cycling 
through the process list again. Besides the idea of attempting to render 
more than one part of a document at a time with multiple threads, 
there's also the more practical reality of adding threads to the parser 
and renderer: multiple simultaneous documents. When dealing with 
Firefox,  IE 7, or Opera 9, it is not uncommon to open multiple 
documents simultaneously. If you have a parser that is not 
multithreaded, you end up with a queue system where each document must 
be processed sequentially, and then rendered sequentially.

Take this example, you open a bookmark that opens 3 tabs automatically. 
Tab 1 opens cnn.com, tab 2 opens slashdot.org, and tab 3 opens an image 
stored on your hard drive. Without being multithreaded, the browser 
might have to wait for cnn.com and slashdot.org to be retrieved, parsed, 
and rendered before it could load the image from your hard disk. Or, 
assuming that your retrieval system is multithreaded, and renders 
whatever is completely retrieved first, you still end up waiting for one 
tab to be completely done before the next can be completed. This example 
obviously isn't completely practical, as no one can read multiple tabs 
simultaneously, but the point is still the same: being multithreaded 
allows multiple documents to be parsed and rendered simultaneously.

As for the messaging system and event driven application design, I agree 
with Mark in that it serves our purposes better. In the same example as 
above, the image in tab 3 has no need to go through a SGML/HTML/XML 
parser. It's an image, and we know it's an image, why do we need to send 
it there? In the design we used with Themis, and will be implementing in 
the renderer and image handler, the image handler will see that an image 
document has been retrieved and will process it for the renderer, and in 
the process notify the renderer (and anyone else that might be 
interested) that the image is ready to use. The renderer can then use 
the image when it's ready for it, which could be right away in the 
example above.

In the cases of tabs 1 and 2 (cnn.com and slashdot.org respectively), 
the images could be added to the rendered output as they become 
available. The reason I state "as they become available" is simple: when 
I browse the web, I'm usually much more interested in the text content 
of a page than the images or other media unless that's what I'm looking 
for. I'd much rather have CNN's content appear so I can start looking 
for articles that I'm interested in while the images are being loaded 
than having to wait for the whole page (images and all) to be 
completely  retrieved first.
> That why I've implemented multithreaded fetching in UZI, so that the 
> rendering time will be (deterministic), max(fetching time + rendering, 
> parsing + CSS mapping + rendering), and the code still remains clear of 
> locking primitives.
>
>   
You more than likely will need to have locking of some sort in this 
design, especially when dealing with Linux. Sending and receiving data 
should always be done under lock conditions when done in a multithreaded 
fashion to prevent corruption. Many of the networking functions are not 
reentrant, and can easily cause data corruption as a result.

Raymond

Re: [Themis-dev] Another code base

[Cyril R. <sta...@la...>]

Mark Hellegers a écrit :
> [...] I disagree with that line of thought. I think the messaging overhead is extremely minimal, even on my old computers.
> You also seem to think that there is only one path from start to wherever
> you want to go, but that is also not true. There can be quite a few parts
> in the system interested in the "next step".
> It is also very easy to just drop in a different module or leave out a
> module you don't need.
> Messaging also makes it easier to queue up on tasks.
>
>   
>
>>> In UZI application part, I'm using a thread pool.
>>> Each thread can execute jobs (like fetching documents from server).
>>> However, from the parsing to the mapping it's a single job.
>>>
>>> I'm more concerned about memory handling than making the parser run in a
>>>       
>> separate thread than the renderer.
>>     
>>> UZI memory allocation all goes to a specific interface (BaseAllocator)
>>>       
>> that can decide to reuse objects and so on.
>>     
>>> There is more performance to gain from a correct memory allocation than
>>>       
> >from multithreading something that is inherently linear and sequential.
>
> That is an interesting approach. Do you have some numbers or something
> like that to back that up ?
> I'm not saying I don't believe you. I'd like to know something more about
> that.
>
> Mark
>   
Well, here are the papers I'm referring about:
http://www.cs.umass.edu/%7Eemery/pubs/berger-oopsla2002.pdf  (about 
using an allocator can speed up the allocation time by 40%)
And the older one:
http://www.cs.umass.edu/%7Eemery/pubs/berger-pldi2001.pdf (a basic 
example on heap layers)

I have nothing about threading a parser. However, speaking about data 
flow, I would say that:
  For the rendering to be done, you need the entire DOM tree (just think 
of absolutely positioned or floating elements for one). You also need 
all the linked resources (whatever, CSS, IMG, etc...)
  For the DOM tree to be entire, you need HTML source to be entire.
  For the CSS resource, you also need HTML source to be entire (as, even 
if <style> tag is only allowed in <head>, it's very frequent to find 
them in the body, google is a nice example of that).
 
So yes, surely, you can start parsing an HTML stream, and start making a 
(partial) DOM tree from it, and still start rendering it, all at the 
same time.
However, if there is a table in the page, you'll break everything not 
waiting for table end (as table size is computable only when received 
entirely), and table based layout at clearly too frequent to be ignored.
For example, a <p> tag inside a <table> tag should be rendered before 
the table (it's illegal, but that's how Mozilla and IE does).
If there is some JS, then you'll also have to get the whole things 
before rendering as JS usually deals with DOM tree (so it has to be 
entire if you don't want your JS engine to fail)
Finally, rendering prematuraly usually gives the flashing effect you had 
on Firefox 1.0 when the CSS didn't loaded in time (the <entire> DOM tree 
rendered, then the CSS arrived, and the <entire> DOM tree rerendered, 
moving things around).

Anyway, I don't get the interest of multithreading the parser & renderer 
(mainly for complexifying the parser, because of the multiple locks 
required to avoid data contention) if you are not going to render the 
final page until all steps are done.
On single core CPU, you'll never get any improvment (well, dataflow 
analysis), and on multicore CPU, the locking mechanism (and bad cache 
data) might simply break any gain you're expecting (as, at best, the 
rendering time will be : max(parser time (you need the whole document 
here), fetching time (way longer that anything else), CSS-DOM mapping, 
renderer) so it's likely = to fetching time)

That why I've implemented multithreaded fetching in UZI, so that the 
rendering time will be (deterministic), max(fetching time + rendering, 
parsing + CSS mapping + rendering), and the code still remains clear of 
locking primitives.

Anyway, you might be interested in:
http://weblogs.mozillazine.org/hyatt/archives/2005_05.html
about CSS rule mapping.

Have a nice day,
Cyril

Re: [Themis-dev] Another code base

[Mark H. <ma...@fi...>]

>Hi all,

Hi

[snip]

>So I finally started to look more closely into Themis by march 07. I
>checked out the code. The boost dependency was very unexpected (as it's
>not compiling everywhere, and slow any compilation by a factor of at
>least 2) Other dependencies were cryptlib, and libz. The javascript
>engine looked like those of Mozilla. I then dug into the modules folder;
>and started examining the code down there. The HTML parser used a lot of
>heap allocation (new ...), without obvious object life cycle (where does
>the object goes, is it shared, and so on...). Sure the dev relied on
>boost shared_ptr for their deallocation. The parser also defined it's
>internal constant as char *, so it will be broken with unicode content
>down there. I've checked the fetching code, and it didn't convert to
>UTF-8, so I thought it was WIP. The parser was cleanly structured between
>every kind of elements as a class hierarchy. However, the class hierarchy
>didn't looked that obvious to me.
>
>I would have expected an HTMLParser to be a SGMLParser (as HTML is a part
>of SGML). In place of this, HTMLParser is dependent on BeOS code (why ?),
>deals with OS specific task and message handling (why ?) And in fact it
>doesn't parse anything.

Let me explain a bit about the HTML parser.
It is indeed an SGML parser and it does work.
You can feed it any document, for which you have a DTD and it should
create a DOM tree.
I know we will get trouble with unicode, but we will cross that bridge
when we get there.
The BeOS specific code is only to hook it up to the Themis messaging
system. I have the SGML parser working on my Irix box as well.

>The DOM tree is correct (in fact it's very clean implementation) of DOM
>interfaces. The renderer was inexistant (no rendering at all, in fact, it
>even lacked the CSS box model's algorithm to map DOM nodes to boxes, and
>reflow them).
>
>As I didn't took time to understand the structure of the code, creating a
>renderer from Themis project would have required studying this in
>details, and rewriting a lot of BeOS specific code (even probably
>rewriting the hierarchy).
>
>So finally, I decided to roll my own. I priviledged compliant and memory
>clean code over speed. I started to implement DOM v2.1 interfaces in UZI.
>Then an charset aware SGML parser (even through only UTF8 and latin1 is
>supported for now) Then I added an HTML parser to produce a DOM tree I
>then added an CSS box model mapper (including CSS types parsing, and
>structures), and table rendering I finally implemented a renderer for all
>the CSS boxes spit from the mapper. I've added a CacheEngine with fast
>lookup too.

That is a very impressive amount of functionality.

>My design included allocators from the beginning (meaning there is no
>"new" in the code except in the allocators, and every object is returned
>to the allocator when required). This means that I'll be able to reuse
>returned objects instead of calling the heap again (a page like
>news.google.fr can take up to12MB of memory just for storing the picture
>and text). I also added test vectors from the beginning (I'm amazed to
>see that they are still some many projects around that doesn't even test
>their code correctness), so I'm sure the DOM implementation works 100% as
>expected. The HTML parser works 100% as expected on VALID documents with
>supported charset. The testing the CSS mapping isn't concluant yet (and
>testing them automatically is very difficult), so there is no test vector
>for it, and, as obvious as it may seems, most of the issue I had (and
>still have) come from that part exactly.

Ah, testing. Yes, that is always a bit of a forgotten part of most
projects.
I try to do the basic tests, but I haven't gotten around to writing more
extensive tests.

[snip Uzi design]

>Ouf,  here we are. I needed a browser to test the renderer on real world
>code. I started testing sites around, and got very pleasant and
>unpleasant surprises. The hardest part is probably creating a DOM tree
>like the author intended, but correcting the errors the author made.
>There is a DTD for validating documents, but most (if not all) website
>just don't care about this, and almost no website validates the DTD. So I
>had to handle so many errors, I not even sure I'm correct by now.

Yeah, it's probably the fault of the early browsers that just accepted
anything.
I'm hoping to build in good error reporting in the Themis HTML parser, so
website designers using Themis to test their websites can easily spot
mistakes in their design.

>The current state is:   - The browser actually renders web pages, like
>any non CSS browser and allow to browse (well... sortof as form aren't
>POSTed yet)   - I still need to include the CSS parsing code to CSS
>properties   - I still need to correct some rendering error on the CSS
>mapper   - I still need to find a way to map CSS rules to DOM node very
>very fast.   - I still have to choose a Javascript engine, and bind it to
>the DOM interface (my current choice would probably be See).   - I still
>have to write tests for all the above.

That is very impressive.

>I'm amazed to see Themis starting to work again. I don't follow that much
>Haiku, but I do think it's good to have choice. I've met Kevin on SeeDev
>mailing list, and he told me about your great news.

Well, I'm not sure yet wether to pick it up again, but we'll see.

>I'm not going to tell you "trash themis, use UZI's renderer, you'll have
>results in 2 weeks". I'm proud of my code and I suppose you are proud of
>yours.

Depends on which part of the code we are talking about. :)

[snip Kevin's explanation of Themis]

>I answered:
>>I've used this programming scheme before (well, Win32 message
>programming I mean).
>>However I've dropped the idea of message based task management.
>>If you think about this, the application is only always doing the same
>task in the same order (HTML fetching -> HTML parsing -> DOM CSS mapping
>-> CSS box rendering) everytime.
>>This means that you pay the message queue cost between every subtask
>while you really don't need to (no one else is using this message
>anyway).
>>Sure it looks like your application is more modular, but in reality it's
>not, because all modules still depends on each other (structure
>declaration for one when passing objects in messages), and it adds
>useless the OS messaging queue dependency.

I disagree with that line of thought. I think the messaging overhead is
extremely minimal, even on my old computers.
You also seem to think that there is only one path from start to wherever
you want to go, but that is also not true. There can be quite a few parts
in the system interested in the "next step".
It is also very easy to just drop in a different module or leave out a
module you don't need.
Messaging also makes it easier to queue up on tasks.

>
>>In UZI application part, I'm using a thread pool.
>>Each thread can execute jobs (like fetching documents from server).
>>However, from the parsing to the mapping it's a single job.
>>
>>I'm more concerned about memory handling than making the parser run in a
>separate thread than the renderer.
>>UZI memory allocation all goes to a specific interface (BaseAllocator)
>that can decide to reuse objects and so on.
>>There is more performance to gain from a correct memory allocation than
>from multithreading something that is inherently linear and sequential.

That is an interesting approach. Do you have some numbers or something
like that to back that up ?
I'm not saying I don't believe you. I'd like to know something more about
that.

Mark

-- 
Spangalese for beginnners:
`Wiggilo wagel hoggle?'
`Where can I scrub my eyeballs?'

Re: [Themis-dev] Some changes

[Raymond C. R. <z3r...@bb...>]

Fredrik Modéen wrote:
> How would this work? Are this information about my browser position when
> saving the bookmark?
>
>   
It would be relatively simple, the filename might be the name from the 
web page, or if it's media, then the file name on the site. The BFS 
attributes would include the URL, as well as the order number in the 
bookmarks menu. Even folders within the bookmarks menu would be treated 
as bookmark items, and have an order number. Each order number would be 
unique within its relative folder... So for top level folders you might 
have 3 bookmarks and a folder like this:

0 Themis Home
1 Haiku-Os.org
2 BeBits
3 Other Stuff (folder)

The Other Stuff folder might would have it's own 0-indexed list of files 
and folders
0 CNN
1 weather.com
2 Yahoo!

And so on. The only real  work involved would be in editing the ordering 
and making sure the order numbers are unique in each folder.

>> My vote would be favicon as icon as well... But you're right, there's
>> still lots of time to work on bookmarks, and revise things.
>>     
> I like the idea about favicon (after looking it up with wikipedia).
>
> oh and I'm looking at the people application to see how to store
> attributes to a file.
>
>   
If I remember correctly, it's mainly using BNodeInfo or BNode, but it's 
not too difficult. The tricky part is whether or not you set up an index 
for those attributes or not, so that they're query enabled...

Raymond

Re: [Themis-dev] Some changes

[Michael W. <em...@m-...>]

Fredrik Modéen schrieb:
>> Michael Weirauch wrote:
>>> When it comes to storing bookmarks, a file system hierarchy would do,
>>> but at some point one might wan't arrange entries. So a flat filesystem
>>> won't do. I guess that is why Mozilla & Co is going for some crazy
>>> netscape bookmark html. Filesystem plus display-position attribute,
>>> favicon as attrib or icon anyone?
>>>
>> I like the file system idea with a display-position attribute, myself.
> How would this work? Are this information about my browser position when
> saving the bookmark?
The bookmarks are not sorted alphabetically if the user desires. (By 
default they just would be.) In the bookmark manager, the user might 
change the sequence they are displayed from top to bottom. I very much 
like structuring my bookmarks in FireFox' Bookmarks Toolbar folders.
But hey, it's just an attribute and a sort criterion. Nothing to mind 
about to seriously right now.

>> My vote would be favicon as icon as well... But you're right, there's
>> still lots of time to work on bookmarks, and revise things.
> I like the idea about favicon (after looking it up with wikipedia).
> 
> oh and I'm looking at the people application to see how to store
> attributes to a file.

Yo, the people application is a nice intro into attribute storage, iirc. 
And you can screw all your contacts with a console-cp-backup. Funny if 
you realize that a little to late and already deleted them at the 
original place ;)

Greetings,
Michael
> 
>> Raymond
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2005.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> Themis-dev mailing list
>> The...@li...
>> https://lists.sourceforge.net/lists/listinfo/themis-dev
>>
> 
> 

Re: [Themis-dev] Some changes

[Fredrik <fr...@mo...>]

> Michael Weirauch wrote:
>> When it comes to storing bookmarks, a file system hierarchy would do,
>> but at some point one might wan't arrange entries. So a flat filesystem
>> won't do. I guess that is why Mozilla & Co is going for some crazy
>> netscape bookmark html. Filesystem plus display-position attribute,
>> favicon as attrib or icon anyone?
>>
>
> I like the file system idea with a display-position attribute, myself.
How would this work? Are this information about my browser position when
saving the bookmark?

> My vote would be favicon as icon as well... But you're right, there's
> still lots of time to work on bookmarks, and revise things.
I like the idea about favicon (after looking it up with wikipedia).

oh and I'm looking at the people application to see how to store
attributes to a file.

>
> Raymond
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Themis-dev mailing list
> The...@li...
> https://lists.sourceforge.net/lists/listinfo/themis-dev
>


-- 
MVH
Fredrik Modéen

Re: [Themis-dev] Some changes

[Fredrik <fr...@mo...>]

> Fredrik Modéen wrote:
>> When It comes to bookmarks/history I think we should use NetPositive's
>>
>> I know we talked about not use NetOptimist (from now NetOpt) code but
>> when
>> I checked there code I found some gui part's that I couldn't resist
>> using.
>>
>> I think that we should use gui part from NetOpt and later when we want
>> to
>> rewrite the gui we can remove them. (I can make big remarks around them
>> :)
>> )
>>
>> Any way here are some copy/modify/enhanced code from NetOpt
>>
>> Now working are
>> - Fullscreen (not using boolean NetOpt did we are using
>> SetMarked/IsMarked
>> in menu) - Show bookmarks (from NetOpt) will soon remove and use
>> Themis/bookmarks - List bookmarks (this one are a big copy from NetOpt)
>> to
>> see if it works start NetPositive and save bookmarks
>>
>> Not working fully but will be easy to fix when you can brows with Themis
>> -
>> Drag/Drop a NetPositive bookmark and get the URL
>> - Forward, Backward, Home these have now a menu option and arrows should
>> work.
>>
>> Will work on next
>> - Get a print window pop up don't know what else it should send. (Not
>> from
>> NetOpt) - Save bookmark (Not from NetOpt)
>>
>> The only thing left from NetOpt that can perhaps be used are Proxy
>> settings  (don't know if it can)
>>
>> One thing I would like to do are follow BeBook and Haiku way of coding.
>> Changing win.cpp to ThemisWin, app.cpp to ThemisApp and so on. Also
>> changing some Methods (don't remember the c++/c name for the same) to
>> start with uppercase letter.
>>
>> all of this are only in the framwork directory and commondef.h
>>
> I'm going to be taking a look at your changes in a few minutes, after I
> finish installing BeOS Max in Virtual PC again... It seems that my old
> BeOS Max VPC installation has disappeared... possibly due to my wife...
> possibly due to some drive cleaning that I did a few weeks ago. The
> latter is unlikely since I know I've run BeOS since then... My work was
> lost but yours won't be... :-)
I hop it wasn't alot of work you lost.

Have a good and authorial (don't know if this are the word I'm looking
for) look as I think that I may have broke something.
http://www.bebits.com don't give me any text in the browser.

>
> Raymond
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Themis-dev mailing list
> The...@li...
> https://lists.sourceforge.net/lists/listinfo/themis-dev
>


-- 
MVH
Fredrik Modéen

Re: [Themis-dev] Some changes

[Fredrik <fr...@mo...>]

> Fredrik Modéen schrieb:
>>> Fredrik Modéen schrieb:
>>>>>> When It comes to bookmarks/history I think we should use
>>>>>> NetPositive's
>>>>>>
>>>>>> I know we talked about not use NetOptimist (from now NetOpt) code
>>>>>> but
>>>>>> when
>>>>>> I checked there code I found some gui part's that I couldn't resist
>>>>>> using.
>>>>>>
>>>>>> I think that we should use gui part from NetOpt and later when we
>>>>>> want
>>>>>> to
>>>>>> rewrite the gui we can remove them. (I can make big remarks around
>>>>>> them
>>>>>> :)
>>>>>> )
>>>>> Pending a response from the devs of Themis, I'd suspect that would be
>>>>> a
>>>>> good way to go....  Just make sure you do this on all the pieces you
>>>>> pull
>>>>> in from NetOptimist.  Then, later Mark or Raymond can note what they
>>>>> want
>>>>> done.  You mentioned rewriting the code later..., did you have a
>>>>> better
>>>>> design in mind that meant you might want to write the UI code from
>>>>> scratch
>>>>> according to that design?
>>>> No  most clean up but Raymond talked about rewriting the gui :)
>>> Why not taking any NetOptimist code as inspiration and implementing our
>>> own? This way you don't scew the code with comments and code which
>>> might
>>> be adopted to the existing ui framework and be later removed or
>>> refactored anyway. Sounds like work done twice. Nobody hinders you/us
>>> to
>> It would be work done twice :) and I have no problem with both :)
>
> I was just arguing for not replicating work. But hey, I don't mind at
> all helping out with the adopted code if I once get a usable development
> system ;)
And I'm all for that :)

>
>>> take inspiration from other code. And btw, it doesn't hurt if the
>>> Bookmarks implementation takes a little longer as there is nothing to
>>> show your rendered bookmarks right now.
>> Right now nothing happens but years ago I think one was able to surf the
>> net?
>> Are this anything we can do to show anything? like the html text?
>
> http://themis.m-phasis.de/shots/Themis_20040314_1.png ;) So still a long
> way to go.
k then I have the same result now :)

>
> Michael
>
>>> When it comes to storing bookmarks, a file system hierarchy would do,
>>> but at some point one might wan't arrange entries. So a flat filesystem
>>> won't do. I guess that is why Mozilla & Co is going for some crazy
>>> netscape bookmark html. Filesystem plus display-position attribute,
>>> favicon as attrib or icon anyone?
>> About file bookmarks as netposetive does, makes them search able from
>> the
>> OS taking advantage of BFS. If wee need more information in a file then
>> we
>> can add those.
>>
>>> Greets,
>>> Michael
>>>
>>>>> I don't know if this would be appropriate or not, but maybe add a
>>>>> TODO
>>>>> file to the main Themis directory and make a note in there that there
>>>>> is
>>>>> code with the weird comments around it that need to be cleaned up or
>>>>> dealt
>>>>> with at some point (so it isn't forgotten about).  Or if you think
>>>>> this
>>>>> is
>>>>> redundant or a bad idea, then don't bother.
>>>> To me the code from NetOpt are not a bad thing it's more what Raymond
>>>> says
>>>> and accepts in to svn :)
>>>>
>>>> I have looked at UZI but having problem compiling it. Will take that
>>>> in
>>>> the UZI project and some other questions like images are handled and
>>>> so
>>>> on.
>>>>
>>>>
>>>>> Kevin
>>>>>
>>>>> -------------------------------------------------------------------------
>>>>> This SF.net email is sponsored by: Microsoft
>>>>> Defy all challenges. Microsoft(R) Visual Studio 2005.
>>>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>>>> _______________________________________________
>>>>> Themis-dev mailing list
>>>>> The...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/themis-dev
>>>>>
>>>>
>>>
>>> -------------------------------------------------------------------------
>>> This SF.net email is sponsored by: Microsoft
>>> Defy all challenges. Microsoft(R) Visual Studio 2005.
>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>> _______________________________________________
>>> Themis-dev mailing list
>>> The...@li...
>>> https://lists.sourceforge.net/lists/listinfo/themis-dev
>>>
>>
>>
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Themis-dev mailing list
> The...@li...
> https://lists.sourceforge.net/lists/listinfo/themis-dev
>


-- 
MVH
Fredrik Modéen

Re: [Themis-dev] Some changes

[Raymond C. R. <z3r...@bb...>]

Michael Weirauch wrote:
> When it comes to storing bookmarks, a file system hierarchy would do, 
> but at some point one might wan't arrange entries. So a flat filesystem 
> won't do. I guess that is why Mozilla & Co is going for some crazy 
> netscape bookmark html. Filesystem plus display-position attribute, 
> favicon as attrib or icon anyone?
>   

I like the file system idea with a display-position attribute, myself. 
My vote would be favicon as icon as well... But you're right, there's 
still lots of time to work on bookmarks, and revise things.

Raymond