I have roughly 7,000 records which total about 109MB, rather small I suppose. While doing ./openwebspider -i somesite.com -s -e -x -l 100, I found that is would not do a true cache, it would only display the UnHtml returned text. Then when doing a search, it would take about 3 or 4 seconds to return the results. So problem number 1. Found that in openwebspider-0.7.c, the -x feature has no code in it.
Just shows "case 'x':" no code. Would not keep the html in the cache version of the website/page.
case 'x'"
xCacheHtml=1;
Should fix it, unless I have completly missed something. :) possible.
Okay so now that I have explained this. My search results took longer still yet.
Found that, and this is the negative side, if you remove lines 81 through 87. Which eliminates the "count" of items found. My speed of results returned went to 0.022 to about 0.231ish.
So from around 4 to 6 seconds down to less then half a second I think is pretty good.
I hope this helps others and maybe someone can tell me if what I did is really wrong, stupid, or just insane.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Had to do a little more testing. This idea will only return how many results are allowed by the php script. Specifically the results per page lines. On the other hand if you have a small site, this idea wouldn't be bad.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This could be negative or positive.
I have roughly 7,000 records which total about 109MB, rather small I suppose. While doing ./openwebspider -i somesite.com -s -e -x -l 100, I found that is would not do a true cache, it would only display the UnHtml returned text. Then when doing a search, it would take about 3 or 4 seconds to return the results. So problem number 1. Found that in openwebspider-0.7.c, the -x feature has no code in it.
Just shows "case 'x':" no code. Would not keep the html in the cache version of the website/page.
case 'x'"
xCacheHtml=1;
Should fix it, unless I have completly missed something. :) possible.
Okay so now that I have explained this. My search results took longer still yet.
Found that, and this is the negative side, if you remove lines 81 through 87. Which eliminates the "count" of items found. My speed of results returned went to 0.022 to about 0.231ish.
So from around 4 to 6 seconds down to less then half a second I think is pretty good.
I hope this helps others and maybe someone can tell me if what I did is really wrong, stupid, or just insane.
Had to do a little more testing. This idea will only return how many results are allowed by the php script. Specifically the results per page lines. On the other hand if you have a small site, this idea wouldn't be bad.