ebiness-crawler Mailing List for EbiNess

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi all,

I've put out a new release (v0.2) that's got a whole bunch of changes in
it. Most of it related to stability and memory leaks...

cya
Mike

On Sun, 3 Jun 2001, Sellaro wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Sat, 2 Jun 2001, Mike Davis wrote:
>
> > Sellaro and Allan, could you guys start working on a storage system that
> > we can plug into the existing crawler? We've spoken about a few things,
> > namely B trees, ReiserFS and using hashing algos... I think we should go
> > with Sellaro's plan use some form of B tree, if you guys are happy with
> > that? The basic interface should be something like the 'StorageAdapter'
> > class in the 'include/storageadapter.h' file.
>
> Firts of all, I would like to apologize for my long silence. I have just
> been contracted to work on a connectivity organization here in Brazil. I'm
> spending some time to organize myself there. Sorry.
>

Busy man... I fully understand :-)

>
> About storage on Ebiness, sounds great to implement something with
> B-trees. As a matter of fact, Allan is working on a B-Tree mechanism for
> his thesis (correct me if I'm wrong, Mr. Reffson ;). We could take a look
> at his classes, adapt part of his code and integragrate it with Ebiness.
>

Allan, what area of B-trees are you working on? Anything cool?

You guys think you could get something done in the next few weeks?
I've been looking at how I can make the changes and I'll be doing soem
work this week.

cya
Mike

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi folks

I've just found an interesting article at Web reference about visual
search engines. I think they've got a point :) Check it out at
http://www.webreference.com/new/010607.html#feature

- --
Sellaro

Network Administrator
Federal University of Cear=E1 (UFC)
Computer Science Dept.
PCT-Motorola Laboratory

PGP/GNU PG Public Key Available at http://sellaro.pathbot.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE7INcVx3CBMgQUSLYRAiWBAJ9JvtyO55BHnQjuMgK3YCmGUJPPtgCfSXdf
ax8B04vGaZ2GBb87RrDkKQI=3D
=3DRZVZ
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sat, 2 Jun 2001, Mike Davis wrote:

> Sellaro and Allan, could you guys start working on a storage system tha=
t
> we can plug into the existing crawler? We've spoken about a few things,
> namely B trees, ReiserFS and using hashing algos... I think we should g=
o
> with Sellaro's plan use some form of B tree, if you guys are happy with
> that? The basic interface should be something like the 'StorageAdapter'
> class in the 'include/storageadapter.h' file.

Firts of all, I would like to apologize for my long silence. I have just
been contracted to work on a connectivity organization here in Brazil. I'=
m
spending some time to organize myself there. Sorry.

About storage on Ebiness, sounds great to implement something with
B-trees. As a matter of fact, Allan is working on a B-Tree mechanism for
his thesis (correct me if I'm wrong, Mr. Reffson ;). We could take a look
at his classes, adapt part of his code and integragrate it with Ebiness.

- --
Sellaro

Federal University of Cear=E1 (UFC)
Computer Science Dept.
PCT-Motorola Laboratory

PGP/GNU PG Public Key Available at http://sellaro.pathbot.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE7GoZxx3CBMgQUSLYRAs9fAJ4s74dsdv8VhX5kGQS/naqg4DAHQACdHdK6
GaUN2LKeZqcG7rhJ2paDtpM=3D
=3Df/4u
-----END PGP SIGNATURE-----

Hi guys,

Well, time to finally get down to some dirty work!

Sellaro and Allan, could you guys start working on a storage system that
we can plug into the existing crawler? We've spoken about a few things,
namely B trees, ReiserFS and using hashing algos... I think we should go
with Sellaro's plan use some form of B tree, if you guys are happy with
that? The basic interface should be something like the 'StorageAdapter'
class in the 'include/storageadapter.h' file.

Mari, I like that idea about using lexx, although I've never played with
it - could you have a play? Basically all we need is something that can
parse an Xml document into a tree (this includes xHtml), and also be able
to handle Html's silliness, like the <p> tag that doesn't get closed...
It's another thing that we can plug into the existing crawler and do
comparitive tests.

I'll have a go at removing the database calls and replacing them with
calls to a StorageAdapter. In the meantime, I'll make it use a Gdbm file
as the back end.

Mike

  ebi...@li... wrote: 
Send Ebiness-crawler mailing list submissions to
ebi...@li...

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.sourceforge.net/lists/listinfo/ebiness-crawler
or, via email, send a message with subject or body 'help' to
ebi...@li...

You can reach the person managing the list at
ebi...@li...

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Ebiness-crawler digest..."

Today's Topics:

1. Re: decent Html parser? (Allan Reffson Granja Lima)

--__--__--

Message: 1
Date: Mon, 28 May 2001 11:13:12 -0300 (BRT)
From: Allan Reffson Granja Lima 
To: Mike Davis 
cc: ebi...@li...
Subject: Re: [Ebiness-crawler] decent Html parser?

Can be a template in C++ to recognize a "Regular expression" like href="">'?
Looking for "regex++"...

--------------------------------------------
Allan Reffson Granja Lima (al...@li...) 
ICQ: 34004301 
Mestrado em Ciencia da Computacao 

On Sat, 26 May 2001, Mike Davis wrote:

> Hi guys,
> 
> Well, I've already written an parser that simply extracts ' tags from Html for the crawler, but was wondering if anyone knows of a> good, stable parser that will generically parse Html? I think this would> be far superior.> > I think the 'expat' library (hosted on SF - http://expat.sourceforge.net/)> is the standard for Xml parsing, so we can use that for any Xml/xHtml we> come across.> > Mike> > > _______________________________________________> Ebiness-crawler mailing list> Ebi...@li...> http://lists.sourceforge.net/lists/listinfo/ebiness-crawler> --__--__--_______________________________________________Ebiness-crawler mailing lis...@li...://lists.sourceforge.net/lists/listinfo/ebiness-crawlerEnd of Ebiness-crawler Digest">'
> tags from Html for the crawler, but was wondering if anyone knows of a
> good, stable parser that will generically parse Html? I think this would
> be far superior.
> 
> I think the 'expat' library (hosted on SF - http://expat.sourceforge.net/)
> is the standard for Xml parsing, so we can use that for any Xml/xHtml we
> come across.
> 
> Mike
> 
> 
> _______________________________________________
> Ebiness-crawler mailing list
> Ebi...@li...
> http://lists.sourceforge.net/lists/listinfo/ebiness-crawler
> 

It is easy to write Parsers in lex (lexical analyser) come with Linux or UNIX.

It is easy to develop and output will be a C program satisfying parser reqmts.

I can do this, If you could tell me what exactly has to be parsed.

mari.

--__--__--

_______________________________________________
Ebiness-crawler mailing list
Ebi...@li...
http://lists.sourceforge.net/lists/listinfo/ebiness-crawler

End of Ebiness-crawler Digest

" When you do nothing, Nothing works. "

---------------------------------
Do You Yahoo!?
Yahoo! Mail Personal Address - Get email at your own domain with Yahoo! Mail.
Can be a template in C++ to recognize a "Regular expression" like <a
href="">'?
Looking for "regex++"...

--------------------------------------------
Allan Reffson Granja Lima (al...@li...) 
ICQ: 34004301 
Mestrado em Ciencia da Computacao 

On Sat, 26 May 2001, Mike Davis wrote:

> Hi guys,
> 
> Well, I've already written an parser that simply extracts '<a href="">'
> tags from Html for the crawler, but was  wondering if anyone knows of a
> good, stable parser that will generically parse Html? I think this would
> be far superior.
> 
> I think the 'expat' library (hosted on SF - http://expat.sourceforge.net/)
> is the standard for Xml parsing, so we can use that for any Xml/xHtml we
> come across.
> 
> Mike
> 
> 
> _______________________________________________
> Ebiness-crawler mailing list
> Ebi...@li...
> http://lists.sourceforge.net/lists/listinfo/ebiness-crawler
> 

On Fri, 25 May 2001, Mari Kannan wrote:

>
>   Hi Mike,
> I volunteer for this Networking service part. Im not able to open the link you sent for the
>
> latest design doument. So please send that as attachment, if there is any problem with the link.
>
> As I said earlier, Im getting time once in  a week to browse my mails. Im not sure, if that pace
>
> is going to be OK.
>
> Mari.
>
>

Hi Mari,

That pace should be fine. Are there any areas which you'd like to discuss?

What I'd like to do is get the storage system sorted out, for as Sellaro
has pointed out, it's the most important bit of the crawler... then we
can do some test runs with the crawler as it's currently structured. Once
we've got a significant amount of data (I've got a 60Gb disk waiting for
this purpose), we can then start working on the browser in parallel with
getting the crawler into a better state.

After all, the most interesting bit (and the whole point) of the project
is the 3D bit :-)

Mike

>
>   ebi...@li... wrote: Send Ebiness-crawler mailing list submissions to
> ebi...@li...
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.sourceforge.net/lists/listinfo/ebiness-crawler
> or, via email, send a message with subject or body 'help' to
> ebi...@li...
>
> You can reach the person managing the list at
> ebi...@li...
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Ebiness-crawler digest..."
>
>
> Today's Topics:
>
> 1. v0.2 design doc (Mike Davis)
>
> --__--__--
>
> Message: 1
> Date: Sat, 19 May 2001 23:20:11 +0100 (BST)
> From: Mike Davis
> To: ebi...@li...
> cc: md...@ki...
> Subject: [Ebiness-crawler] v0.2 design doc
>
> Hi all,
>
> Apologies for the silence, but as I mentioned previously, I'm in the
> middle of getting a new contract!
>
> Anyway, I've put together a design document for the next version of the
> crawler, which should detail some of the ideas I'm hoping we can put
> together. You can get it at http://ebiness.sf.net/docs/design/v0_2.html.
>
> So, please have a read and comment as much as you can - the more input,
> the better. Also bear in mind that it's not at all finished. I'll be
> putting time in on this one tommorow and during the week (with your
> additions I hope!). If anyone feels a particular interest in an area
> (Sellaro, Storage ;-), take it...
>
> Also, with regards to the general documentation, I'll be finishing that
> next week as well. I hope it helped some of you. All of this documentation
> will be put into cvs on Monday.
>
> So, next week we can hopefully get moving.
>
> cya
> Mike
>
>
>
>
> --__--__--
>
> _______________________________________________
> Ebiness-crawler mailing list
> Ebi...@li...
> http://lists.sourceforge.net/lists/listinfo/ebiness-crawler
>
>
> End of Ebiness-crawler Digest
>
> " When you do nothing, Nothing works. "
>
>
> ---------------------------------
> Do You Yahoo!?
> Yahoo! Auctions $2 Million Sweepstakes  -  Got something to sell?

Hi,

It would appear that Sourceforge has yanked the ebiness.sf.net domain,
which was supposed to replace ebiness.sourceforge.net...
Anyway, the latter still works, so you can get the docs at:
http://ebiness.sourceforge.net/docs/

I think they may be having some problems :-/

cya
Mike

  Hi Mike,
I volunteer for this Networking service part. Im not able to open the link you sent for the

latest design doument. So please send that as attachment, if there is any problem with the link.

As I said earlier, Im getting time once in  a week to browse my mails. Im not sure, if that pace

is going to be OK. 

Mari.

  ebi...@li... wrote: Send Ebiness-crawler mailing list submissions to
ebi...@li...

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.sourceforge.net/lists/listinfo/ebiness-crawler
or, via email, send a message with subject or body 'help' to
ebi...@li...

You can reach the person managing the list at
ebi...@li...

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Ebiness-crawler digest..."

Today's Topics:

1. v0.2 design doc (Mike Davis)

--__--__--

Message: 1
Date: Sat, 19 May 2001 23:20:11 +0100 (BST)
From: Mike Davis 
To: ebi...@li...
cc: md...@ki...
Subject: [Ebiness-crawler] v0.2 design doc

Hi all,

Apologies for the silence, but as I mentioned previously, I'm in the
middle of getting a new contract!

Anyway, I've put together a design document for the next version of the
crawler, which should detail some of the ideas I'm hoping we can put
together. You can get it at http://ebiness.sf.net/docs/design/v0_2.html.

So, please have a read and comment as much as you can - the more input,
the better. Also bear in mind that it's not at all finished. I'll be
putting time in on this one tommorow and during the week (with your
additions I hope!). If anyone feels a particular interest in an area
(Sellaro, Storage ;-), take it...

Also, with regards to the general documentation, I'll be finishing that
next week as well. I hope it helped some of you. All of this documentation
will be put into cvs on Monday.

So, next week we can hopefully get moving.

cya
Mike

--__--__--

_______________________________________________
Ebiness-crawler mailing list
Ebi...@li...
http://lists.sourceforge.net/lists/listinfo/ebiness-crawler

End of Ebiness-crawler Digest

" When you do nothing, Nothing works. "

---------------------------------
Do You Yahoo!?
Yahoo! Auctions $2 Million Sweepstakes  -  Got something to sell?
Hi guys,

Well, I've already written an parser that simply extracts '<a href="">'
tags from Html for the crawler, but was  wondering if anyone knows of a
good, stable parser that will generically parse Html? I think this would
be far superior.

I think the 'expat' library (hosted on SF - http://expat.sourceforge.net/)
is the standard for Xml parsing, so we can use that for any Xml/xHtml we
come across.

Mike

Hi everybody!

Firstly, I'm sorry for long term without "my introduction"!
I'm Allan Lima, friend of Sellaro, and I've been work a lot in master
thesis... The thesis is about classify, in subjects, www pages in
automatic way...

About the B-tree storage, I've been working in this data
structure. Balanced Tree seams good, mainly because we have Riserfs! But
we have to think a little about the cost of operations insert/delete in a
balaced tree... But for while we can use this... :)

--------------------------------------------
Allan Reffson Granja Lima (al...@li...) 
ICQ: 34004301 
Mestrado em Ciencia da Computacao 

On Fri, 25 May 2001, Mike Davis wrote:

> Hi,
> 
> Sellaro, you were talking about implementing a B-tree solution for
> storage... well why don't we use a filesystem that's build with that kind
> of thing in mind:
> http://www.namesys.com/
> 
> ReiserFs uses a balanced tree for the filesystem, so it would, in theory,
> work the same way. We would then have to write an interface to it, but
> that would eliminate a whole pile of work.
> 
> We wouldn't have to write tree checkers/repair/creation/maintenance progs,
> for when the tree got nuked or whatever.
> 
> What do you think?
> 
> Also, I haven't heard much from anyone for a while - are you guys still
> there? :-)
> 
> I'm hoping to finish off the crawler docs soon and start working on the
> browser docs next week. Can you guys tell me who's interested in which
> particular area? I know I want to do both the infrastructure side of the
> crawler and the 3D bit of the browser.
> 
> Obviously, as the browser hasn't even been looked at, it's open for
> suggestions on all aspects. So far I've looked at SDL and OpenGl for the
> general 3D side of things and as far as I'm concerned, we only need to do
> a Linux (and Solaris) version of the browser, but if anyone is a whiz at
> MFC/DirectX, maybe we could get something going for WinXX?
> 
> cya
> Mike
> 
> 
> 
> _______________________________________________
> Ebiness-crawler mailing list
> Ebi...@li...
> http://lists.sourceforge.net/lists/listinfo/ebiness-crawler
> 

Hi,

Sellaro, you were talking about implementing a B-tree solution for
storage... well why don't we use a filesystem that's build with that kind
of thing in mind:
http://www.namesys.com/

ReiserFs uses a balanced tree for the filesystem, so it would, in theory,
work the same way. We would then have to write an interface to it, but
that would eliminate a whole pile of work.

We wouldn't have to write tree checkers/repair/creation/maintenance progs,
for when the tree got nuked or whatever.

What do you think?

Also, I haven't heard much from anyone for a while - are you guys still
there? :-)

I'm hoping to finish off the crawler docs soon and start working on the
browser docs next week. Can you guys tell me who's interested in which
particular area? I know I want to do both the infrastructure side of the
crawler and the 3D bit of the browser.

Obviously, as the browser hasn't even been looked at, it's open for
suggestions on all aspects. So far I've looked at SDL and OpenGl for the
general 3D side of things and as far as I'm concerned, we only need to do
a Linux (and Solaris) version of the browser, but if anyone is a whiz at
MFC/DirectX, maybe we could get something going for WinXX?

cya
Mike

Hi Guys,

I was wondering if anyone has had the time to read the design doc? If so,
it would be great if you guys could criticise it, talk about bits you woud
like to see added, altered or removed or whatever...

Also, I haven't actually finished the docs properly, but something I'd
like to put in there is the publish/subscribe bus for the services.

This would be a network-based (and shared memory for local
services) system, whereby each service must register with the bus when
starting up. The registration would include stating the service's version
info, it's dependencies and it's capabilites.

The idea would be that we could scale the system by (inter)connecting
buses. A possibility is to use Corba to do all the networking etc and
simply build some classes on top of that.

An alternative is to use some form of object migration, where an object
can move to a service, process data on the spot, then take what it needs
to another service. This is probably the most exciting way to do it :-)

Any other ideas?

Mike

Hi all,

Apologies for the silence, but as I mentioned previously, I'm in the
middle of getting a new contract!

Anyway, I've put together a design document for the next version of the
crawler, which should detail some of the ideas I'm hoping we can put
together. You can get it at http://ebiness.sf.net/docs/design/v0_2.html.

So, please have a read and comment as much as you can - the more input,
the better. Also bear in mind that it's not at all finished. I'll be
putting time in on this one tommorow and during the week (with your
additions I hope!). If anyone feels a particular interest in an area
(Sellaro, Storage ;-), take it...

Also, with regards to the general documentation, I'll be finishing that
next week as well. I hope it helped some of you. All of this documentation
will be put into cvs on Monday.

So, next week we can hopefully get moving.

cya
Mike

Hi ppl,

Another one here to lend his hand! Came to know about the proj on =
sourceforge. Think it's cool... and hope it'll be a success.

more about me:

Friends call me vish. I stay in Fremont, CA. I have 3 years of =
experience in C/C++ on UNIX mainly designing and developing network =
management applications. But that's my job. My personal inclination is =
systems programming and graphics programming. Designed and developed a =
3D home designer which won a national level award in India. Have =
experience writing fast 3D animation code for VGA adapter.

Right now hoping to brainstorm a bit.. and do something good and =
interesting.

Cheers,
vish

Hi guys,

I've managed to write some stuff about the crawler. It's not finished (by
a long way), but it does describe some of the more important areas for
development. It's all here:
http://ebiness.sourceforge.net/docs/index.html

and the Ebiness main page is here:
http://ebiness.sourceforge.net/

Also, I hope you guys have downloaded the existing code from sf and had a
look? You can get it here if not:
https://sourceforge.net/project/showfiles.php?group_id=12535

Any comments and help would be welcome. I apologise in advance - my Html
skills aren't that wonderfull. If anyone feels like writing a stylesheet,
that would be great... :-)

cya
Mike

Hi Mike,

Thanks for the intro as well.

Yes please document when ever you get free time. As I said I would be interested in doing Networking stuff. So let us make it a big success.

Mari.

" When you do nothing, Nothing works. "

---------------------------------
Do You Yahoo!?
Yahoo! Auctions $2 Million Sweepstakes  -  Got something to sell?
hi folks,
=20
           =20
              Looks like i am the most inexperienced among all of us. =
This is my first entry to open source community.
I have almost 3 years experience in development application on windows =
platform. and nil in other OS as i have learnt only as my syllabus while =
doing some course ( I haven't done any projects as such in  OS other =
than windows ). I am a computer science graduate from Mangalore =
University, India.
I am currently working on Wince based devices with EVC as front end =
tool.
I have also worked a GDBM( Win32 port ) but ditched after i found as I =
found Toolchest as the solution.
ummm the other reason was i tampered with the code so much that .. i =
found no return . again the reason ..  no documentation as i went on =
modifying the GDBM source .=20
            As kannan told , i have been trying hard to understand and =
grasp as much as i can from every mail mike and  sellaro wrote,
Unfortunately,  ;-( I don't have the Idea of how the whole system works =
as i haven't yet been able to find the complete document for it .
=20
Regards
Sushruth
=20

 =20

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi there folks

Following the guidelines of our modest community, I should introduce
myself :)

My name is Alisson Sellaro (gender: male... believe, here in Brazil,
Alisson is a male name ;) but you all can call me Sellaro. I am a senior
Computer Science student at Federal University of Ceara, Brazil.

I am currently a partial time network manager of Eldorado Research
Institute Laboratory, a Motorola research institute. I've spent all my
graduation working on Network Management and Distributed Systems, with
enfasis in Database. Mybe this is the reason I've received the title of
"Storage-man"  by Mike ;)

My SourceForge username is Sellaro. If you have some time to check out th=
e
projects I'm included, you'll see three of my research projects here and
Ebiness.

I hope I can be usefull, maske some new developer friends and type some
linecodes with ya all ;)

Oh, before I forget, please, let's take a little of our times and try to
fill out each of our parties SourceForge avaliations so we get a more
trustable developers score. I don't like the idea to be classified as a
number, but sometimes they help other people to reach us.

See you all

- --
Sellaro

Network Administrator
Federal University of Cear=E1 (UFC)
Computer Science Dept.
PCT-Motorola Laboratory

PGP/GNU PG Public Key Available at http://sellaro.pathbot.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE6/+fJx3CBMgQUSLYRAioaAJ44py+auD3Q9W0YkOI/q/HgKMsrMQCfc2he
L8HKMDZSE6tFWKxSGIQkXi4=3D
=3DwJ5L
-----END PGP SIGNATURE-----

Hi,

Could you also mention in your intro what your SF user names are? 

Thanks
Mike

Hi all,

On Sun, 13 May 2001, Mari Kannan wrote:

<snip>

> Mike, I have a request for you. Why dont u document your
> requirements or idea in a word document or text file and
> keep it updated. Because I found it difficult to dig the
> whole story. The initial mails are archived and when i 
> tried to access it said the server was getting backedup so 
> cant access for an hour or so.
> 
> Not only for that, tracking some things in mail going to be 
> tuff. I dont mind volunteering for this work. But as I dont 
> know the things from starting, no way I can do it.
> 
> So please put all your ideas and if the requirement could 
> be documented that also and mail it to the list?
> 

Yup, that sounds like a good plan... I was going to do something like
this, Sellaro suggested using the task manager, which is probably a good
idea - that's what SF is all about, right? :-)

Also, I think it's a good idea to start using the documentation manager
and the http://ebiness.sourceforge.net website to manage docs and
requirements.

And thanks for the intro - could everyone else please introduce
yourselves? It would be good for everyone to know who's who, what your
capabilites are and what you're interested in!

I'll start with myself:

I'm Mike Davis, lived in both Canada and South Africa, now living in the
UK.

At the moment I've been doing some Perl work for an Sports Media company,
which is about to go bust, which is why I've been so quiet for
the past week or so. My main area of interest and skill is on the
C++/Unix/Networking side of things, which makes this project quite
interesting for me. I'm also quite keen to get to know OpenGL and graphics
programming in general.

I've done tons of other languages and played with lots of
technologies, but I won't bore you with the details. Most of my work
has involved the Internet in some way or other, which I've worked with
since 1994, having been a BBS freak before that.

The project was inspired by the incredible poster 'The Whole Internet',
done by Peacock Maps (http://www.peacockmaps.com/). I thought I'd go a bit
further and build a system that would allow someone to look at the way
different sites interlink, in 3D. 

So far it's been a year and a bit (only started the SF site in September)
and now I think it's time to get things moving along a bit faster - that's
where you guys come in and hopefully we can build something nice and cool
and not take another year to finish! :-)

Anyway, as I mentioned before, the company I'm currently working for is
possibly closing down, so I've decided to take some time off (two weeks)
and focus on various things, including getting another contract, but
mainly this project... so expect a lot of mail and please excuse me if I
get a bit distracted sometimes. 

The way I'd like to work it is to split the tasks up into a few big pieces 
and then hand them off to people. Sellaro has agreed to be the 'Storage'
person, and hopefully the rest of you will be interested in a
particular area enough to take charge of it... or just participate if you
haven't got enough time.

cya
Mike

Hi folks,

Hope everybody is doing great. Mike said that it would be 
good to introduce myself to everybody in the list. 

Education:
Im from India and have done BS (or called as BE in India)
in Computer Science and Engg. from a reputed college.

work experience:
Passed out in June  1998 and worked from July 1998 for 
SAP labs India. Then right now with Deluxe group of 
companies in US which have a branch out here in India.

All work experience has mostly been in Windows 
environment(unfortunately).

Came to know about sourceforge.net from my colleage who is 
also interested in doing some thing interesting.

Both of us are trying to design and develop a generic frame 
work which would have classes for command useful stuffs 
like security, searching, transactional based database 
works, which could be used ready-made for any of our future 
projects in our company.

I subscribed to this list, so that it would feed me for the 
week-ends when I get bored. Mailed Mike asking some info
about Crawler as it read C++ resource wanted.

I check my mails once in a week as right now no net 
connection at home.

Mike, I have a request for you. Why dont u document your
requirements or idea in a word document or text file and
keep it updated. Because I found it difficult to dig the
whole story. The initial mails are archived and when i 
tried to access it said the server was getting backedup so 
cant access for an hour or so.

Not only for that, tracking some things in mail going to be 
tuff. I dont mind volunteering for this work. But as I dont 
know the things from starting, no way I can do it.

So please put all your ideas and if the requirement could 
be documented that also and mail it to the list?

Thanks,
Mari.

" When you do nothing, Nothing works. "

---------------------------------
Do You Yahoo!?
Yahoo! Auctions $2 Million Sweepstakes  -  Got something to sell?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 8 May 2001, Mike Davis wrote:

> Ummm, would be interesting, if you've got the time... What was your exa=
ct
> strategy going to be? B-Tree based system?

In fact, a B* or B+-tree.

> It seems to me that you're quite interested in the storage side and as
> such, I hereby delgate that aspect of the project to you ;->

Uhmmm. OKay. I accept the delegation ;)

Back to pre-hash algorithm, I would like to know if you have already
planned something. Have you thinked about key size? Hash algorithm? My
first impression was to use MD5 or some other similar algorithm, but I
noticed that their generated key size could affect somohow filesystem
performance. What do you think?

> So I think what I will do is organise a list of things that need doing =
and
> we can talk about them and then split them up amongst us (the intereste=
d
> parties). At the moment, I'm quite interested in pushing on and startin=
g
> work on the browser portion of the project, as we've now got a
> semi-functional crawler and quite a bit of data to play with.

Take your time. You could use SF's task manager. It really works! :)

- --
Sellaro

Network Administrator
Federal University of Cear=E1 (UFC)
Computer Science Dept.
PCT-Motorola Laboratory

PGP/GNU PG Public Key Available at http://sellaro.pathbot.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE6+Nx3x3CBMgQUSLYRAvL5AJ9hgbVdEvXBIy6+HdOQEYAWqFl3fACeKx8Q
OLMdfByrWZoEBx6R5XfyNrY=3D
=3DBqUA
-----END PGP SIGNATURE-----

On Thu, 3 May 2001, Sellaro wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Tue, 1 May 2001, Mike Davis wrote:
> 

<snip>

> > The reason I suggest using gdbms is purely because they do work well
> > and reliably when their size is below 100Mb and also we haven't got masses
> > of time to spend on writing something from scratch (that's already been
> > done anyway!)
> 
> It is a very delicated point. If we choose to use gdbm (even with the
> pre-hash algorithm) we may be delimiting a future expansion boundary. I
> suggest we try to implement both strategies (simplified) and to take a
> look at some benchmarks.
> 

Ummm, would be interesting, if you've got the time... What was your exact
strategy going to be? B-Tree based system?

It seems to me that you're quite interested in the storage side and as
such, I hereby delgate that aspect of the project to you ;->

> Is this too burocratic?
> 

Not sure, sounds reasonable, but we'll have to be carefull we don't waste
too much time, otherwise we'll still be here next year discussing the same
stuff!

From some of the other projects I've been involved with, it seems
important to keep things moving along, even if it's just a slow pace,
otherwise everyone gets real bored and dissappears...

So I think what I will do is organise a list of things that need doing and
we can talk about them and then split them up amongst us (the interested
parties). At the moment, I'm quite interested in pushing on and starting
work on the browser portion of the project, as we've now got a
semi-functional crawler and quite a bit of data to play with.

BTW, apologies for taking so long to respond - went on a bit of a holiday
:-)

Mike

2000	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct (1)	Nov	Dec
2001	Jan	Feb	Mar	Apr (8)	May (22)	Jun (4)	Jul (1)	Aug	Sep	Oct	Nov	Dec

ebiness-crawler Mailing List for EbiNess

ebiness-crawler — Information &amp; discussion relating to the crawler portion of the project

ebiness-crawler — Information & discussion relating to the crawler portion of the project