themis-dev — Development list.

Re: [Themis-dev] CSS support

[Mark H. <ma...@fi...>]

>On Wed, Feb 18, 2009 at 10:21 PM, Raymond C. Rodgers <
>ra...@ba...> wrote:
>
>> Not yet, it's on my list for this weekend. I managed to solve some more
>> cookie problems about 10 days ago that I haven't submitted yet. Most
>> notably, the date parsing that I had been using seemed to suddenly stop
>> working, and Haiku has a simple to use date parser that is in a MIT
license
>> that I'll probably selectively compile in for BeOS support, which means
I'll
>> need to track down the correct file to re-appropriate. In the process
of
>> tracking down another cookie related bug, I had to go digging through
some
>> of the sources for the cache, and got to re-familiarize myself with it.
The
>> problem is probably something simple like accidentally assigning the
cache
>> token instead of comparing it or something like that... Nonetheless,
I'll be
>> digging into that on Saturday.
>>
>
>Ok, the problem looks like it may be related to the gzip encoding
support,
>which I think should be rewritten at some point. I noticed that with the
>encoding enabled, it added some extra characters to the beginning of the
>file transferred, but when the support was removed, the file was saved
>correctly. Right now, it's easier to comment out the line of code that
adds
>the accept encoding header to the HTTP request than to go through and
>rewrite the gzip support, so I'm going to do that.

Hi,

I just tried it again with the changes you made, but I think it still
happens.
I tried themis.sf.net and what I can see from the html parser, is that it
gets a bunch of tags after the last html tag. I looked in the cached file
and they don't seem to be there either.
What does Themis do on your system if you try that site ?

Mark

-- 
Spangalese for beginnners:
`Parut d'psuhe.'
`My hair-do is on fire.'

Re: [Themis-dev] CSS support

[Raymond R. <ra...@ba...>]

On Wed, Feb 18, 2009 at 10:21 PM, Raymond C. Rodgers <
ra...@ba...> wrote:

> Not yet, it's on my list for this weekend. I managed to solve some more
> cookie problems about 10 days ago that I haven't submitted yet. Most
> notably, the date parsing that I had been using seemed to suddenly stop
> working, and Haiku has a simple to use date parser that is in a MIT license
> that I'll probably selectively compile in for BeOS support, which means I'll
> need to track down the correct file to re-appropriate. In the process of
> tracking down another cookie related bug, I had to go digging through some
> of the sources for the cache, and got to re-familiarize myself with it. The
> problem is probably something simple like accidentally assigning the cache
> token instead of comparing it or something like that... Nonetheless, I'll be
> digging into that on Saturday.
>

Ok, the problem looks like it may be related to the gzip encoding support,
which I think should be rewritten at some point. I noticed that with the
encoding enabled, it added some extra characters to the beginning of the
file transferred, but when the support was removed, the file was saved
correctly. Right now, it's easier to comment out the line of code that adds
the accept encoding header to the HTTP request than to go through and
rewrite the gzip support, so I'm going to do that.

As for the double notification about the page being completed... I haven't
looked into it yet, but I suspect that the HTTP add-on is sending the
initial notification, and then the SiteManager is turning around and sending
the same or similar message.

Raymond

Re: [Themis-dev] HTML parser hang...

[Raymond C. R. <ra...@ba...>]

Raymond C. Rodgers wrote:
>>  
> You're probably right, though I suppose could also be something in the 
> cache code; perhaps it's not providing all of the data to the parser 
> for some reason. I really wish I had a system capable of running R5 to 
> compare and test with. :-/ I'll zip up the directory later and post it 
> online; please [try to] compile it and see if you run into any 
> problems with it when you get a chance. That will eliminate or verify 
> that my code changes are the cause.
>
> Thanks Mark!
Ok, I lied a bit... I committed my work in a branch so that I could 
continue to work on it and yet test out some changes without fear of 
loss. The branch is at /branches/haiku_test/ and may or may not compile 
on R5 with the makefile I included. I know it's missing a couple of 
files that I reappropriated from Haiku for the parsedate function, but 
I'll get those in there in a minute or two.

Raymond

Re: [Themis-dev] HTML parser hang...

[Raymond C. R. <ra...@ba...>]

Mark Hellegers wrote:
>> I was running some tests to see if I could find a sign of the error that
>> Mark was referring to about the HTML parser getting the wrong content,
>>     
> and
>   
>> ran into some odd behavior on a fairly simple page I wrote specifically
>>     
> for
>   
>> simple testing: http://www.dmsfriend.com/themistest.html . Basically, the
>> parser would get the content and begin going through it, but would stop
>> suddenly without error or message until the window was closed, at which
>> point it would continue.
>>
>> I'm attaching two files to this message. The first is hang_point.txt
>>     
> which
>   
>> contains the output immediately preceding the hang, including the spot
>>     
> where
>   
>> it hung in the middle of outputting some text. The second is
>>     
> all_output.txt
>   
>> which contains all of the output from that execution, including the hang,
>> and the output generated when the window was closed.
>>
>> The easiest way to sync up the two files is to do a search for "Time
>>     
> taken
>   
>> for loading the schema: 1.270000" in each and then scroll to the end of
>>     
> the
>   
>> output in hang_point.txt.
>>
>> The Themis build was on Haiku r29147 built on 2/7/2009 (a build I did
>> myself), with only the cache, http, and html parser add-ons. Admittedly,
>>     
> the
>   
>> code used still does have some uncommitted changes in the http add-on but
>>     
> I
>   
>> don't think they're related to this hanging issue.
>>     
>
> Hi Raymond,
>
> I tested the website on an R5 system and it parses it without problems
> here.
> It seems to stop in the middle of a printf call on your system, which is
> strange.
> If it keeps happening consistently there is probably a bug in Haiku.
>
>
>   
You're probably right, though I suppose could also be something in the 
cache code; perhaps it's not providing all of the data to the parser for 
some reason. I really wish I had a system capable of running R5 to 
compare and test with. :-/ I'll zip up the directory later and post it 
online; please [try to] compile it and see if you run into any problems 
with it when you get a chance. That will eliminate or verify that my 
code changes are the cause.

Thanks Mark!
Raymond

Re: [Themis-dev] CSS support

[Raymond C. R. <ra...@ba...>]

Mark Hellegers wrote:
>> Still up for doing an irc chat?
>>     
>
> Yes, but I won't be at home the next two weekends, so it will have to be
> after that.
> I think I have time in the weekend after that (7 or 8 of march).
>
>
>   
That works for me, just let me know when.

Raymond

Re: [Themis-dev] HTML parser hang...

[Mark H. <ma...@fi...>]

>I was running some tests to see if I could find a sign of the error that
>Mark was referring to about the HTML parser getting the wrong content,
and
>ran into some odd behavior on a fairly simple page I wrote specifically
for
>simple testing: http://www.dmsfriend.com/themistest.html . Basically, the
>parser would get the content and begin going through it, but would stop
>suddenly without error or message until the window was closed, at which
>point it would continue.
>
>I'm attaching two files to this message. The first is hang_point.txt
which
>contains the output immediately preceding the hang, including the spot
where
>it hung in the middle of outputting some text. The second is
all_output.txt
>which contains all of the output from that execution, including the hang,
>and the output generated when the window was closed.
>
>The easiest way to sync up the two files is to do a search for "Time
taken
>for loading the schema: 1.270000" in each and then scroll to the end of
the
>output in hang_point.txt.
>
>The Themis build was on Haiku r29147 built on 2/7/2009 (a build I did
>myself), with only the cache, http, and html parser add-ons. Admittedly,
the
>code used still does have some uncommitted changes in the http add-on but
I
>don't think they're related to this hanging issue.

Hi Raymond,

I tested the website on an R5 system and it parses it without problems
here.
It seems to stop in the middle of a printf call on your system, which is
strange.
If it keeps happening consistently there is probably a bug in Haiku.

Mark

-- 
Spangalese for beginnners:
'Blu shef farn wahr'
'May I please have my leg back'

[Themis-dev] HTML parser hang...

[Raymond R. <ra...@ba...>]

Computing Schema
Computing empty set
Time taken: 0.027000
Computing first set
Doing a compute first round...
Time taken for round: 0.449000
Doing a compute first round...
Time taken for round: 0.194000
Doing a compute first round...
Time taken for round: 0.176000
Time taken: 0.819000
Time taken for loading the schema: 1.270000
Starting to scan the HTML document
Loaded the document
Got first token: <
Rule declaration contains HTML
Tag HTML is correct here.
Rule content has token HEAD
Rule content contains HEAD
Looking in normal content
Rule , contains HEAD
Parsing token with rule 0 in sequence part
Rule HEAD contains HEAD
Rule declaration contains HEAD
Tag HEAD is correct here.
Rule content has token TITLE
Rule content contains TITLE
Looking in normal content
Rule & contains TITLE
Parsing token with rule 0 in all part
Rule TITLE contains TITLE
Rule declaration contains TITLE
Tag TITLE is correct here.
Rule content has token #PCDATA
Found text
Rule content contains #PCDATA
Parsing text in content
Looking in normal content
Found text
Rule #PCDATA contains #PCDATA
Text is correct here.
Not doing anything with this token here
Ending parse rule #PCDATA
Content found correctly in parseContent
Looking for any extra exceptions
Not doing anything with this token here
Ending parse rule content
Content found correctly
Found correct end tag: TITLE
Not doing anything with this token here
Ending parse rule declaration
Not doing anything with this token here
Ending parse rule TITLE
Ending parse rule &
Content was not correct in parseContent
Looking for any extra exceptions
Ending parse rule content
Content was not correct
Found correct end tag: HEAD
Rule not found for start tag BODY
Ending parse rule declaration
Rule not found for start tag BODY
Ending parse rule HEAD
Parsing token with rule 1 in sequence part
Rule BODY contains BODY
Rule declaration contains BODY
Tag BODY is correct here.
Rule content has token TABLE
Rule content contains TABLE
Looking in normal content
Rule | contains TABLE
Rule TABLE contains TABLE
Rule declaration contains TABLE
Tag TABLE is correct here.
Rule content has token TR
Rule content contains TR
Looking in normal content
Rule , contains TR
Skipping optional sequence rule
Skipping optional sequence rule
Skipping optional sequence rule
Skipping optional sequence rule
Parsing token with rule 4 in sequence part
Rule TBODY contains TR
Rule declaration contains TR
Skipping optional start tag.
Rule content has token TR
Rule c

Re: [Themis-dev] CSS support

[Mark H. <ma...@fi...>]

>
>Still up for doing an irc chat?

Yes, but I won't be at home the next two weekends, so it will have to be
after that.
I think I have time in the weekend after that (7 or 8 of march).

Mark

-- 
Spangalese for beginnners:
'Blu shef farn wahr'
'May I please have my leg back'

Re: [Themis-dev] CSS support

[Raymond C. R. <ra...@ba...>]

Mark Hellegers wrote:
> I also learned a lot since then, especially from the rewrite of the html
> parser.
> My intention is to first get the css parser to work only parsing the
> files. Putting them in a nice structure can come later. I really need to
> take a good look good at that, because I don't want to rewrite this part
> three times as well. :)
>
>   
Good idea! :D
>> On a separate but tangentially related subject, some time soon, I'm going
>> to start piecing together a central activity tree which will house
>> information about URLs loaded within a certain lifetime (say the last 4-5
>> hours). Basically, it'll list URLs directly requested, the images and
>> other media files requested as a result of parsing the page's HTML,
>> content statuses, sizes, and types, and possibly window/tab
>> relationships, and window/tab history. I still have a lot of thought to
>> put into this, and I hope to actually write out a simple design document
>> before I actually start coding it. The basic idea is the URL manager that
>> I think I had Michael begin on years ago, but a bit better thought out
>> than what I had described at the time... :-) I know it might be best to
>> reuse code or complete what was already started, but I want to start from
>> scratch on it.
>>     
>
> That would be really cool. I'm looking forward to your ideas on this one.
> Did you manage to take a look at the problem with reading the wrong data ?
>
>   
Not yet, it's on my list for this weekend. I managed to solve some more 
cookie problems about 10 days ago that I haven't submitted yet. Most 
notably, the date parsing that I had been using seemed to suddenly stop 
working, and Haiku has a simple to use date parser that is in a MIT 
license that I'll probably selectively compile in for BeOS support, 
which means I'll need to track down the correct file to re-appropriate. 
In the process of tracking down another cookie related bug, I had to go 
digging through some of the sources for the cache, and got to 
re-familiarize myself with it. The problem is probably something simple 
like accidentally assigning the cache token instead of comparing it or 
something like that... Nonetheless, I'll be digging into that on Saturday.

Still up for doing an irc chat?

Raymond

Re: [Themis-dev] CSS support

[Mark H. <ma...@fi...>]

>Mark Hellegers wrote:
>> Hi,
>>
>> I mentioned I was going to look into the state of the CSS support in
>> Themis.
>> I have done some investigation and it seems we can use the parser that
>is
>> already in there.
>> I'm going to start working on making it work and hooking it into Themis
>> properly.
>>
>> Parsing, however, is only part of the whole CSS picture. We need a
>proper
>> structure to store the CSS information in and the CSS parser has some
>> support, but very minimal. I started something long ago, but never
>> finished it and I'm not yet sure if I can use that or if that needs to
>be
>> scrapped.
>> On top of that, the rules to determine which rule applies to an element
>> can be fairly complex as far as I understand it, so I will tackle that
>> part after parsing is properly working.
>>
>> Any comments, suggestions, etc are always welcome.
>>
>> Mark
>>
>>
>   I remember a few discussions (on which I think we disagreed :) )
>about trying to integrate CSS into the DOM tree some how, possibly
>applying the attributes directly to the branches. I think I've learned
>enough about CSS since then to see that it might not be practical;
>basically supporting your position about it not being a good idea.
>(Although I think you disagreed on the principle of modularity.) Beyond
>that, I don't really have any thoughts on CSS. :-/

I also learned a lot since then, especially from the rewrite of the html
parser.
My intention is to first get the css parser to work only parsing the
files. Putting them in a nice structure can come later. I really need to
take a good look good at that, because I don't want to rewrite this part
three times as well. :)

>On a separate but tangentially related subject, some time soon, I'm going
>to start piecing together a central activity tree which will house
>information about URLs loaded within a certain lifetime (say the last 4-5
>hours). Basically, it'll list URLs directly requested, the images and
>other media files requested as a result of parsing the page's HTML,
>content statuses, sizes, and types, and possibly window/tab
>relationships, and window/tab history. I still have a lot of thought to
>put into this, and I hope to actually write out a simple design document
>before I actually start coding it. The basic idea is the URL manager that
>I think I had Michael begin on years ago, but a bit better thought out
>than what I had described at the time... :-) I know it might be best to
>reuse code or complete what was already started, but I want to start from
>scratch on it.

That would be really cool. I'm looking forward to your ideas on this one.
Did you manage to take a look at the problem with reading the wrong data ?

Mark

-- 
Spangalese for beginnners:

Re: [Themis-dev] CSS support

[Raymond C. R. <ra...@ba...>]

Mark Hellegers wrote:
> Hi,
>
> I mentioned I was going to look into the state of the CSS support in
> Themis.
> I have done some investigation and it seems we can use the parser that is
> already in there.
> I'm going to start working on making it work and hooking it into Themis
> properly.
>
> Parsing, however, is only part of the whole CSS picture. We need a proper
> structure to store the CSS information in and the CSS parser has some
> support, but very minimal. I started something long ago, but never
> finished it and I'm not yet sure if I can use that or if that needs to be
> scrapped.
> On top of that, the rules to determine which rule applies to an element
> can be fairly complex as far as I understand it, so I will tackle that
> part after parsing is properly working.
>
> Any comments, suggestions, etc are always welcome.
>
> Mark
>
>   
I remember a few discussions (on which I think we disagreed :) ) about 
trying to integrate CSS into the DOM tree some how, possibly applying 
the attributes directly to the branches. I think I've learned enough 
about CSS since then to see that it might not be practical; basically 
supporting your position about it not being a good idea. (Although I 
think you disagreed on the principle of modularity.) Beyond that, I 
don't really have any thoughts on CSS. :-/

On a separate but tangentially related subject, some time soon, I'm 
going to start piecing together a central activity tree which will house 
information about URLs loaded within a certain lifetime (say the last 
4-5 hours). Basically, it'll list URLs directly requested, the images 
and other media files requested as a result of parsing the page's HTML, 
content statuses, sizes, and types, and possibly window/tab 
relationships, and window/tab history. I still have a lot of thought to 
put into this, and I hope to actually write out a simple design document 
before I actually start coding it. The basic idea is the URL manager 
that I think I had Michael begin on years ago, but a bit better thought 
out than what I had described at the time... :-) I know it might be best 
to reuse code or complete what was already started, but I want to start 
from scratch on it.

Raymond

[Themis-dev] CSS support

[Mark H. <ma...@fi...>]

Hi,

I mentioned I was going to look into the state of the CSS support in
Themis.
I have done some investigation and it seems we can use the parser that is
already in there.
I'm going to start working on making it work and hooking it into Themis
properly.

Parsing, however, is only part of the whole CSS picture. We need a proper
structure to store the CSS information in and the CSS parser has some
support, but very minimal. I started something long ago, but never
finished it and I'm not yet sure if I can use that or if that needs to be
scrapped.
On top of that, the rules to determine which rule applies to an element
can be fairly complex as far as I understand it, so I will tackle that
part after parsing is properly working.

Any comments, suggestions, etc are always welcome.

Mark

-- 
Spangalese for beginnners:
`Lidbush don uenen deksez.'
`The Library is full of tar.'

Re: [Themis-dev] Rewrite of HTML parser

[Raymond R. <ra...@ba...>]

Mark Hellegers wrote:
> Haha, that's no problem.
> I checked in a number of changes over the weekend to make the parser a lot
> better. It can actually parse quite a few of my test files and the chances
> of getting into an infinite loop have been reduced quite a bit.
> The problem with the data for a file doesn't seem to be in the cache
> addon. I had a quick look and it does seem that the extra bytes are
> actually stored in the cache.
>
> I still have a few problems to fix in the parser, but for testing the rest
> of Themis it is far enough I think.
> I'd like to have a look at where the css support stands right now. That
> might take a while as I don't know what exactly is there, but I'll report
> back when I know what needs to be done.
>
> Is it a good idea to have a chat on irc maybe next weekend or so, to
> discuss what we want to do and maybe have a quick session of looking at
> the most serious bugs that are currently in Themis ?
>
> Mark
>
>   
Having a chat sounds like a good idea, though I don't think I'll be able 
to do it for another two weeks or so (Feb 21 or 22). Is that ok with 
you? In the mean time, could you do me a favor? Will you see if BeOS or 
Zeta (whichever you're running) has a header in the support directory 
called "parsedate.h"? It appears that this may be a Haiku only header, 
but it just solved a problem with the cookie support and I'd like to use 
it if possible. Then again, it's apparently under the MIT license so it 
shouldn't be a problem to include it and the source directly in Themis 
if necessary...

Raymond

Re: [Themis-dev] Rewrite of HTML parser

[Mark H. <ma...@fi...>]

>Mark Hellegers wrote:
>> Hi Raymond,
>>
>> I have this week off from work, so I could spend to time to get the new
>> HTML parser into a usable state. Took quite a bit more effort than I
>> thought. :)
>>
>> I'm seeing some odd things though:
>> - On some sites I get the request to start parsing twice, bu the second
>> time I don't get any data. I built in a check in the parser to prevent
>the
>> second parse, but the second request shouldn't happen. I get this on
>> www.osnews.com for example.
>> - I don't get the complete data for themis.sf.net, although I can see
>that
>> the cache file does have all the data.
>>
>> I might look into it if I have the time, but getting the most serious
>bugs
>> out of the parser has a higher priority right now.
>>
>> Mark
>>
>>   It's probably something broken in either the HTTP add-on or in the
>message system... I think I noticed that with the old HTML parser as
>well, so I believe it's just been cleared of any wrong doing. :-) I'll
>see if I can dig into it either on Saturday or Sunday, just to make sure
>I properly clear your good name. :-)

Haha, that's no problem.
I checked in a number of changes over the weekend to make the parser a lot
better. It can actually parse quite a few of my test files and the chances
of getting into an infinite loop have been reduced quite a bit.
The problem with the data for a file doesn't seem to be in the cache
addon. I had a quick look and it does seem that the extra bytes are
actually stored in the cache.

I still have a few problems to fix in the parser, but for testing the rest
of Themis it is far enough I think.
I'd like to have a look at where the css support stands right now. That
might take a while as I don't know what exactly is there, but I'll report
back when I know what needs to be done.

Is it a good idea to have a chat on irc maybe next weekend or so, to
discuss what we want to do and maybe have a quick session of looking at
the most serious bugs that are currently in Themis ?

Mark

-- 
Spangalese for beginnners:

Re: [Themis-dev] Rewrite of HTML parser

[Raymond C. R. <ra...@ba...>]

Mark Hellegers wrote:
> Hi Raymond,
>
> I have this week off from work, so I could spend to time to get the new
> HTML parser into a usable state. Took quite a bit more effort than I
> thought. :)
>
> I'm seeing some odd things though:
> - On some sites I get the request to start parsing twice, bu the second
> time I don't get any data. I built in a check in the parser to prevent the
> second parse, but the second request shouldn't happen. I get this on
> www.osnews.com for example.
> - I don't get the complete data for themis.sf.net, although I can see that
> the cache file does have all the data.
>
> I might look into it if I have the time, but getting the most serious bugs
> out of the parser has a higher priority right now.
>
> Mark
>
>   
It's probably something broken in either the HTTP add-on or in the 
message system... I think I noticed that with the old HTML parser as 
well, so I believe it's just been cleared of any wrong doing. :-) I'll 
see if I can dig into it either on Saturday or Sunday, just to make sure 
I properly clear your good name. :-)

Raymond

Re: [Themis-dev] Rewrite of HTML parser

[Mark H. <ma...@fi...>]

>Mark Hellegers wrote:
>> Hi all,
>>
>> I just checked in a complete rewrite of the HTML parser.
>> It is still very unstable, but it can already parse some pages
>correctly,
>> for example the google homepage.
>> I don't think it is very useful for anyone to test it, if you don't
>know
>> where to find the problem in the code when a page doesn't parse
>correctly.
>> I know it still gets confused on a lot of pages, causing error
>messages,
>> or worse infinite loops.
>>
>> That said, if you are working on another part of Themis (hint hint ;)
>and
>> you need a particular page parsed correctly, I'll be happy to have a
>look
>> at what the problem is.
>>
>> Mark
>>
>>
>   :) I noticed the devcvs messages earlier. :) I'm just finishing up a
>project for work that's had me tied up for the last two months, so I'll
>be getting back to work myself [again] soon. When I was last working with
>the code in November, I noticed a few bugs in processing set-cookie
>headers on certain sites. (Oddly enough, only on Microsoft owned sites.)
>So while it might not be directly related to HTML parsing, I'll be sure
>to keep an eye on what happens.

Hi Raymond,

I have this week off from work, so I could spend to time to get the new
HTML parser into a usable state. Took quite a bit more effort than I
thought. :)

I'm seeing some odd things though:
- On some sites I get the request to start parsing twice, bu the second
time I don't get any data. I built in a check in the parser to prevent the
second parse, but the second request shouldn't happen. I get this on
www.osnews.com for example.
- I don't get the complete data for themis.sf.net, although I can see that
the cache file does have all the data.

I might look into it if I have the time, but getting the most serious bugs
out of the parser has a higher priority right now.

Mark

--
Spangalese for beginnners:
'Blu shef farn wahr'
'May I please have my leg back'



-- 
Spangalese for beginnners:
'Blu shef farn wahr'
'May I please have my leg back'

Re: [Themis-dev] Rewrite of HTML parser

[Raymond C. R. <ra...@ba...>]

Mark Hellegers wrote:
> Hi all,
>
> I just checked in a complete rewrite of the HTML parser.
> It is still very unstable, but it can already parse some pages correctly,
> for example the google homepage.
> I don't think it is very useful for anyone to test it, if you don't know
> where to find the problem in the code when a page doesn't parse correctly.
> I know it still gets confused on a lot of pages, causing error messages,
> or worse infinite loops.
>
> That said, if you are working on another part of Themis (hint hint ;) and
> you need a particular page parsed correctly, I'll be happy to have a look
> at what the problem is.
>
> Mark
>
>   
:) I noticed the devcvs messages earlier. :) I'm just finishing up a 
project for work that's had me tied up for the last two months, so I'll 
be getting back to work myself [again] soon. When I was last working 
with the code in November, I noticed a few bugs in processing set-cookie 
headers on certain sites. (Oddly enough, only on Microsoft owned sites.) 
So while it might not be directly related to HTML parsing, I'll be sure 
to keep an eye on what happens.


Raymond

[Themis-dev] Rewrite of HTML parser

[Mark H. <ma...@fi...>]

Hi all,

I just checked in a complete rewrite of the HTML parser.
It is still very unstable, but it can already parse some pages correctly,
for example the google homepage.
I don't think it is very useful for anyone to test it, if you don't know
where to find the problem in the code when a page doesn't parse correctly.
I know it still gets confused on a lot of pages, causing error messages,
or worse infinite loops.

That said, if you are working on another part of Themis (hint hint ;) and
you need a particular page parsed correctly, I'll be happy to have a look
at what the problem is.

Mark

-- 
Spangalese for beginnners:
`Lunue d'nent.'
`Your axe is swift Stewardess.'

Re: [Themis-dev] V8 STL Error

[Raymond R. <ra...@ba...>]

Raymond Rodgers wrote:
> I still haven't really dug into the code, but I think the 
> PropertyAttribute and PropertyAttributes classes are both based on 
> templates; it's been a couple weeks since I really looked at it, and 
> even then it was just to see if I could make a quick fix. In any 
> event, I suspect it is a limitation on gcc 2.95, as it doesn't even 
> hesistate during compiling on gcc 4.3.2 under OpenSuSE 11.
I just took a closer look at the V8 documentation regarding building and 
requirements (as hard to find as it is), I just noticed that gcc 4.x is 
indeed required. So, I guess V8 will have to wait and see what happens 
with the possible Haiku gcc4 hybrid... In particular if that will mean 
that gcc 4.x will become available on Haiku with R1 or not. D'oh!

Raymond

Re: [Themis-dev] V8 STL Error

[Raymond R. <ra...@ba...>]

Mark Hellegers wrote:
>> Below is the error that I'm getting when using a very simple makefile to
>>     
> try
>   
>> to build Google's V8 Javascript engine under Haiku at the moment:
>>
>> /Prometheus/dev/v8/src/api.cc: In method `void
>> v8::FunctionTemplate::AddInstancePropertyAccessor(v8::Handle<v8::String>,
>> v8::Handle<v8::Value> (*)(v8::Local<v8::String>, const v8::AccessorInfo
>>     
> &),
>   
>> void (*)(v8::Local<v8::String>, v8::Local<v8::Value>, const
>>     
> v8::AccessorInfo
>   
>> &), v8::Handle<v8::Value>, v8::AccessControl, v8::PropertyAttribute)':
>> /Prometheus/dev/v8/src/api.cc:705: static_cast from
>>     
> `v8::PropertyAttribute'
>   
>> to `PropertyAttributes'
>> /Prometheus/dev/v8/src/api.cc: In method `bool
>> v8::Object::Set(v8::Handle<v8::Value>, v8::Handle<v8::Value>,
>> v8::PropertyAttribute = None)':
>> /Prometheus/dev/v8/src/api.cc:1734: static_cast from
>>     
> `v8::PropertyAttribute'
>   
>> to `PropertyAttributes'
>> make: *** [src/api.o] Error 1
>>     
>
> It looks like the static_cast is failing, but it is hard to tell without
> more information.
> So that would mean it is not an STL issue. It might still be a compiler
> issue, though.
>
> Mark
>
>   
I still haven't really dug into the code, but I think the 
PropertyAttribute and PropertyAttributes classes are both based on 
templates; it's been a couple weeks since I really looked at it, and 
even then it was just to see if I could make a quick fix. In any event, 
I suspect it is a limitation on gcc 2.95, as it doesn't even hesistate 
during compiling on gcc 4.3.2 under OpenSuSE 11.

Raymond

Re: [Themis-dev] V8 STL Error

[Mark H. <ma...@fi...>]

>Below is the error that I'm getting when using a very simple makefile to
try
>to build Google's V8 Javascript engine under Haiku at the moment:
>
>/Prometheus/dev/v8/src/api.cc: In method `void
>v8::FunctionTemplate::AddInstancePropertyAccessor(v8::Handle<v8::String>,
>v8::Handle<v8::Value> (*)(v8::Local<v8::String>, const v8::AccessorInfo
&),
>void (*)(v8::Local<v8::String>, v8::Local<v8::Value>, const
v8::AccessorInfo
>&), v8::Handle<v8::Value>, v8::AccessControl, v8::PropertyAttribute)':
>/Prometheus/dev/v8/src/api.cc:705: static_cast from
`v8::PropertyAttribute'
>to `PropertyAttributes'
>/Prometheus/dev/v8/src/api.cc: In method `bool
>v8::Object::Set(v8::Handle<v8::Value>, v8::Handle<v8::Value>,
>v8::PropertyAttribute = None)':
>/Prometheus/dev/v8/src/api.cc:1734: static_cast from
`v8::PropertyAttribute'
>to `PropertyAttributes'
>make: *** [src/api.o] Error 1

It looks like the static_cast is failing, but it is hard to tell without
more information.
So that would mean it is not an STL issue. It might still be a compiler
issue, though.

Mark

-- 
Spangalese for beginnners:
'Blu shef farn wahr'
'May I please have my leg back'

[Themis-dev] V8 STL Error

[Raymond R. <ra...@ba...>]

Below is the error that I'm getting when using a very simple makefile to try
to build Google's V8 Javascript engine under Haiku at the moment:

/Prometheus/dev/v8/src/api.cc: In method `void
v8::FunctionTemplate::AddInstancePropertyAccessor(v8::Handle<v8::String>,
v8::Handle<v8::Value> (*)(v8::Local<v8::String>, const v8::AccessorInfo &),
void (*)(v8::Local<v8::String>, v8::Local<v8::Value>, const v8::AccessorInfo
&), v8::Handle<v8::Value>, v8::AccessControl, v8::PropertyAttribute)':
/Prometheus/dev/v8/src/api.cc:705: static_cast from `v8::PropertyAttribute'
to `PropertyAttributes'
/Prometheus/dev/v8/src/api.cc: In method `bool
v8::Object::Set(v8::Handle<v8::Value>, v8::Handle<v8::Value>,
v8::PropertyAttribute = None)':
/Prometheus/dev/v8/src/api.cc:1734: static_cast from `v8::PropertyAttribute'
to `PropertyAttributes'
make: *** [src/api.o] Error 1


Any ideas? I suspect this may be a limitation in gcc 2.95.x as this will
compile on OpenSuSE 11 which I think has a 4.x version of gcc.

Raymond

Re: [Themis-dev] 12/5/2008 Status

[Raymond R. <ra...@ba...>]

Mark Hellegers wrote:
> Hi Raymond,
>
>   
Hi Mark! Saw you in some pictures from BeGeistert taken by Nicholas 
Blachford! :-)
>> Earlier this week, I managed to put together a tiny Intel Atom
>> 330 based computer which can boot and run Haiku. Aside from needing to
>> use the VESA video driver, and a quirk that keeps popping up when typing
>> (Haiku seems to think I'm occasionally holding a key down long after I've
>> released it), it seems beyond stable enough to do development on.
>>     
>
> Very nice. I'm still waiting for an official alpha release before I give
> it a spin.
>
> [snip makefile changes for Haiku]
>
> Is your makefile still BeOS compatible then ?
> Did you mention the denpendency generation problem to the Haiku developers
> ?
>
>   
At the moment it is as far as I know. I didn't make any real Haiku 
specific modifications to it. Right now, all of the lines that generate 
or include the dependency files have been commented out, and I'm getting 
the suspicious feeling that Haiku's compiler or make might be smarter 
that that included with BeOS R5: during my tinkering, it seems like only 
the appropriate files are getting recompiled when I make a change. I'm 
not sure if changing a header file will do that properly or not, but 
things seem to be fine with out them. The only other changes I've made 
to the makefile is to replace the references to _APP_ in the add-on 
lines with Themis/Themis, and add -soname=_APP_ to the framework compile 
line.
>> I also had to check-out a copy of the repository via cvs to get a
>> pristine copy of the themis.rsrc resource file to replace the copy that
>> was in the Subversion repository. I'm guessing that when I converted the
>> repository to svn years ago from cvs, that the resource file some how got
>> corrupted in the process, and as a result was preventing xres from adding
>> the resource file to the compiled Themis binary.
>>     
>
> Yeah, I noticed that too. Thanks for fixing it.
>
>   
No problem, I wish I had been able to detect that earlier.
>>  The main problem that
>> I've had outside of compiling is that the add-ons are not loading. I
>> haven't investigated the reason they aren't loading yet, but all of them
>> have compiled successfully, and are located where they are supposed to be
>> located. Hopefully, I'll have the time this weekend to investigate this,
>> and possibly even fix the problem(s).
>>     
>
> Did you manage to fix the problem ?
>
>   
Yep, the problem was that because the add-ons were being linked against 
_APP_, which was the traditional way of allowing add-ons to access the 
host's functions and objects, Haiku was looking for a library called 
_APP_. After I figured that out, I tested with a symlink from the 
framework app to the local lib directory with a filename _APP_ and the 
add-ons loaded right away and worked properly. A bit later, I added the 
-soname=_APP_ to the framework compile line and replaced all other 
references to _APP_ with Themis/Themis, and everything continued to 
work. I don't know if that will work on BeOS or not, but I suspect it won't.
>> I intend to build a library out of the V8 Javascript engine that was
>> developed as part of Google's Chrome web browser. I have checked out a
>> copy of that repository and tried compiling it with scons (the build
>> software they're using), but haven't gotten it to do much that way,
>> probably because Haiku isn't on their supported list, so it wasn't taken
>> into account in the SControl file. So, I've started looking for a way to
>> circumvent this by building a makefile, but that's so far resulted in an
>> error dealing with a static cast from one type to another using STL. I'm
>> not very knowledgeable about STL, so this has halted my initial progress
>> in that direction. I don't know if it's a genuine error in the code, or
>> if it's just that gcc 2.95.x which is in use on BeOS and Haiku is too old
>> to handle it. If you happen to know and/or are more knowledgeable about
>> STL than I am, I'd appreciate a bit of help there.
>>     
>
> I program using the STL myself, but I also know the error messages the
> comiler can generate if things go wrong with STL and they are not pretty.
> You can post the error here or maybe on bedevtalk and see what happens.
>
>   
I'll post it here a bit later, I'm under some pressure with my job to 
get a project usable by tomorrow morning.
>> That about covers things for the moment, but please let me know if you
>> have any thoughts, comments, or ideas.
>>     
>
> It's nice to see someone committing code for Themis again. It's been a
> while.
> Are you planning to do real development again ?
>
> Mark
>   
That's my plan, though it won't be the near constant work that we did 
years ago... I have a lot to re-learn, and I'm finding some interesting 
bugs that I'm going to have to figure out. For instance, I discovered 
that there are sites (like Microsoft's live.com) out there that aren't 
sending cookies in the correct format: some aren't sending the domain or 
path that the cookie is supposed to apply to, and while I wrote code to 
handle those situations years ago, it doesn't look like it's working 
properly, so I'm going to have to dig in and find out why not.

Raymond

Re: [Themis-dev] 12/5/2008 Status

[Raymond R. <ra...@ba...>]

Mark Hellegers wrote:
> Very nice. I'm still waiting for an official alpha release before I give
> it a spin.
>
> [snip makefile changes for Haiku]
>
> Is your makefile still BeOS compatible then ?
> Did you mention the denpendency generation problem to the Haiku developers
> ?
>   
(Forgot to answer a question :-) ) No, I haven't mentioned the 
dependency problem to the Haiku developers yet, though I actually did 
search their development mailing list archive earlier today for that 
problem. I'll probably bring it up in the next couple of days.

Raymond

Re: [Themis-dev] 12/5/2008 Status

[Mark H. <ma...@fi...>]

>Hi folks,

Hi Raymond,

> Earlier this week, I managed to put together a tiny Intel Atom
>330 based computer which can boot and run Haiku. Aside from needing to
>use the VESA video driver, and a quirk that keeps popping up when typing
>(Haiku seems to think I'm occasionally holding a key down long after I've
>released it), it seems beyond stable enough to do development on.

Very nice. I'm still waiting for an official alpha release before I give
it a spin.

[snip makefile changes for Haiku]

Is your makefile still BeOS compatible then ?
Did you mention the denpendency generation problem to the Haiku developers
?

> I also had to check-out a copy of the repository via cvs to get a
>pristine copy of the themis.rsrc resource file to replace the copy that
>was in the Subversion repository. I'm guessing that when I converted the
>repository to svn years ago from cvs, that the resource file some how got
>corrupted in the process, and as a result was preventing xres from adding
>the resource file to the compiled Themis binary.

Yeah, I noticed that too. Thanks for fixing it.

>  The main problem that
>I've had outside of compiling is that the add-ons are not loading. I
>haven't investigated the reason they aren't loading yet, but all of them
>have compiled successfully, and are located where they are supposed to be
>located. Hopefully, I'll have the time this weekend to investigate this,
>and possibly even fix the problem(s).

Did you manage to fix the problem ?

> I intend to build a library out of the V8 Javascript engine that was
>developed as part of Google's Chrome web browser. I have checked out a
>copy of that repository and tried compiling it with scons (the build
>software they're using), but haven't gotten it to do much that way,
>probably because Haiku isn't on their supported list, so it wasn't taken
>into account in the SControl file. So, I've started looking for a way to
>circumvent this by building a makefile, but that's so far resulted in an
>error dealing with a static cast from one type to another using STL. I'm
>not very knowledgeable about STL, so this has halted my initial progress
>in that direction. I don't know if it's a genuine error in the code, or
>if it's just that gcc 2.95.x which is in use on BeOS and Haiku is too old
>to handle it. If you happen to know and/or are more knowledgeable about
>STL than I am, I'd appreciate a bit of help there.

I program using the STL myself, but I also know the error messages the
comiler can generate if things go wrong with STL and they are not pretty.
You can post the error here or maybe on bedevtalk and see what happens.

> That about covers things for the moment, but please let me know if you
>have any thoughts, comments, or ideas.

It's nice to see someone committing code for Themis again. It's been a
while.
Are you planning to do real development again ?

Mark

--
Spangalese for beginnners:




-- 
Spangalese for beginnners:
`Parut d'psuhe.'
`My hair-do is on fire.'