icecrawler-devel — devel mailing-list

[Icecrawler-devel] Project membership reset operation

[Guillaume C. <las...@gm...>]

Hi All.

In an effort to keep the project members list clean and to grant full access
only to regular contributers, sourceforge.net accounts for users considered
to have opted out the ICECrawler project, or who are being unable to
participate in it for now, will have their official membership removed today
and wiki permission will be ajusted accordingly.

Sourceforge project membership will now be granted on a case by case basis,
considering needs and advantages both for the applicant and the project.

Please note that it does not concern wiki or development mailing-list
membership. Both are public areas, thus where anyone can interact.

User accounts to be reset are :
- ddnetweaver (Damien Dujardin)
- fabien_b (Fabien Bourdoulous)
- loutebi59 (Clémentine Havrez)
- neoflowen (Florian Leclercq)
- ninimouss (Pierre-Jean Baraud)
- pumm (Thibaut Arribe)
- yannot (Yann Maréchal)

Thank you for your comprehension.

Cheers,

Guillaume

Re: [Icecrawler-devel] [retour arriere]Stratégie de Crawl

[Aurelien B. <aur...@et...>]

En fait je pense mettre trompé, une simple stratégie:

Focus page additionneur (pool d’url, scoring)

suffit à prendre en compte le fait de croiser plusieurs fois le lien tout en
utilisant les scores. Par contre il faut continuer à travailler en gardant le
'site' commme unité de base pour la sortie des données (graph/affichage...).

Aurélien


Selon Aurelien Beltrame <aur...@et...>:

>
> Bonjour tout le monde,
>
> vous trouverez ci joint un pdf de quelque page sur des stratégies de crawl.
> J'ai détaillé un paquet de stratégie qui n'ont que peu d'intéret seul mais
> que
> l'on peut moduler pour donner des choses un peu mieux.
> J'en propose une en derniere page. Par contre en fonction de l'implémentation
> ca
> peut donner des trucs assez sale niveau complexité.
>
> J'attends vos remarques/critiques dessus.
> J'ai pas du tout regarder sur le net si des gens ont publié des strats de
> crawl,
> donc il est peut etre possible de trouver d'autre source d'inspiration.
>
> Bonne intersemestre/fin de stage/début de stage.
>
> Aurélien
>

[Icecrawler-devel] Stratégie de Crawl

[Aurelien B. <aur...@et...>]

Bonjour tout le monde,

vous trouverez ci joint un pdf de quelque page sur des stratégies de crawl.
J'ai détaillé un paquet de stratégie qui n'ont que peu d'intéret seul mais que
l'on peut moduler pour donner des choses un peu mieux.
J'en propose une en derniere page. Par contre en fonction de l'implémentation ca
peut donner des trucs assez sale niveau complexité.

J'attends vos remarques/critiques dessus.
J'ai pas du tout regarder sur le net si des gens ont publié des strats de crawl,
donc il est peut etre possible de trouver d'autre source d'inspiration.

Bonne intersemestre/fin de stage/début de stage.

Aurélien

Re: [Icecrawler-devel] (no subject)

[Guillaume C. <las...@gm...>]

Salut,

Techniquement je vois pas ce qui t'emb=EAte avec le changement de get() ver=
s
request(), et je t'avoue ne pas tr=E8s bien comprendre pourquoi tu veux lui
passer un QFile* =E0 la place de buffer_, sachant que la m=E9thode
QHttp::request() demande un QIODevice* et que aussi bien QBuffer que QFile
h=E9ritent de QIODevice... En plus, on a besoin de stocker le contenu dans =
le
buffer, sinon on a aucun moyen de le traiter apr=E8s... Rajouter des fichie=
rs
dans l'=E9quation c'est pas une bonne id=E9e pour le moment. On y viendra
peut-=EAtre quand le crawler traitera des masses de donn=E9es importantes m=
ais
l=E0, =E7a demande du temps qu'on a pas pour faire =E7a proprement, et on a=
 plus
urgent avant de s'y mettre. Et puis, pour l'instant c'est pas ton taf ! ;-)

Sinon dans les headers, c'est pas sur qu'on ait besoin de passer tout =E7a.=
 Et
ce serait cool que tu saches ce =E0 quoi correspond ce que tu passes -_-
Par exemple :
* Content-Type c'est le serveur qui te le renvoie pour sp=E9cifier le type =
de
donn=E9es qu'il te file, tu n'es pas sens=E9 le d=E9finir, puisque tu ne pe=
ux pas
savoir =E0 l'avance ce qu'il va te renvoyer
* Une connexion en keep-alive, c'est bien, sauf que pour le moment c'est pa=
s
impl=E9ment=E9 chez nous, comme tu peux normalement le constater. On risque=
rait
donc de se retrouver avec plusieurs dizaines de connexions en simultan=E9 v=
ers
un seul et m=EAme serveur qui attendrait des ordres qu'on ne lui enverrait
jamais... Je suis pas sur qu'ils aiment, les serveurs, et les admins qu'il =
y
=E0 derri=E8re non plus.
* Le gzip / deflate encoding, c'est super, =E7a nous fait =E9conomiser de l=
a
place dans les transferts HTTP. Mais je suis pas sur qu'on le g=E8re pour l=
e
moment...
* Accept-Language Fr ? Heu... Oui mais non, on parle pas que le fran=E7ais.=
..
Ne rien mettre, c'est mieux vu qu'on accepte tout en d=E9finitive.

Donc voil=E0, juste : fais gaffe. Ne fais que le n=E9cessaire pour le momen=
t, et
surtout comprends ce que tu fais. Si tu veux rajouter des choses et
exp=E9rimenter une fois que le strict n=E9cessaire marche, libre =E0 toi. M=
ais
pour le moment ce n'est pas le cas...

++ et bon courage.

Guillaume



2008/1/12 Fabien Bourdoulous <fab...@gm...>:

> Salut,
>
> une question concernant toujours l'user-agent.
> Dans le ICNetworkHttpHandler on effectue la requ=EAte de la mani=E8re sui=
vante
> :
> currentRequestId =3D http->get(url_.path(), buffer_); // original
>
> Or en int=E9grant maintenant le QHttpRequestHeader, la requ=EAte est fait=
e de
> la mani=E8re suivante :
> currentRequestId =3D http->request(header, buffer_, file_); //TODO
>
> avec file_ d=E9clar=E9 comme QFile* dans ICNetworkHttpHandler.h
> et request d=E9clar=E9e ainsi :  int QHttp::request ( const QHttpRequestH=
eader
> & header, QIODevice * data =3D 0, QIODevice * to =3D 0 )
>
> > Le file_ est-il important ?
>
>
> Concernant les directives du header HTTP, j'ai inclu =E7a pour l'instant =
:
> QHttpRequestHeader header("GET", url_.path());
> header.setValue("Host", url_.host());
> header.setValue ("User-Agent", userAgent_);
> header.setValue("Accept",
> "text/xml,application/xml,application/xhtml+xml,text/html,text/plain,*/*"=
);
> header.setValue("Accept-Language", "Fr");
> header.setValue("Accept-Encoding", "gzip,deflate");
> header.setValue("Accept-Charset", "iso-8859-1,*,utf-8");
> header.setValue("Keep-Alive", "300");
> header.setValue ("Connection", "keep-alive");
> header.setValue("Content-Type", "text/html; charset=3Diso-8859-1");
>
> > Il y a peut =EAtres des directives obsol=E8tes ou manquantes ?
>
>
> bon week end =E0 tous.
>

[Icecrawler-devel] Fwd: Reunion ?

[Guillaume C. <las...@gm...>]

Si on oublie des gens dans la discussion, =E7a risque de pas le faire...

Au fait, c'est pas aussi sens=E9 servir =E0 =E7a une mailing-list ? :-)

++

---------- Forwarded message ----------
From: Guillaume Ceccarelli <las...@gm...>
Date: Jan 10, 2008 5:15 PM
Subject: Re: Reunion ?
To: St=E9phan Gorget <ph...@gm...>
Cc: Fabien Bourdoulous <fab...@gm...>


Hello ! :-)

J'ai plusieurs petites remarques =E0 faire sur ton travail :

* Tout d'abord la remarque de Steph est tr=E8s juste, la m=E9thode setUser(=
) de
QHttp sert =E0 effectuer de l'authentification et tu vas effectivement avoi=
r
besoin de te servir de QHttpRequestHeader pour propager la cha=EEne de
User-Agent dans les en-t=EAtes HTTP envoy=E9es par ICECrawler.
* Passer l'user-agent dans les constructeurs des handlers c'est sale et =E7=
a
complique les choses inutilement. Appeler une m=E9thode setUserAgent une fo=
is
le handler instanci=E9 est probablement mieux. En plus, =E7a t'=E9vites d'a=
voir =E0
toucher =E0 la factory.
* Dans ICNetworkHandler, il faudrait que tu fasse deux petites choses en
plus de rajouter la propri=E9t=E9 _userAgent et la m=E9thode qui permet de =
la
modifier :
    * ajouter une m=E9thode de lecture de ta propri=E9t=E9 : QString
ICNetworkHandler::userAgent() const;
    * ajouter un "Q_PROPERTY(QString userAgent READ userAgent WRITE
setUserAgent)" au d=E9but de la d=E9claration de la classe dans
icnetworkhandler.h
* C'est pas forc=E9ment une bonne id=E9e de changer la valeur par d=E9faut =
dans
ICRequestSettings. La propri=E9t=E9 =E9tant r=E9glable dans l'interface gra=
phique,
c'est =E0 l'utilisateur de choisir s'il a envie de faire passer ICECrawler
pour Firefox. Par contre, on peut proposer une liste de valeurs par d=E9fau=
t
contenant un user-agent firefox, mais apr=E8s c'est un travail =E0 faire au
niveau de l'interface graphique.

Sinon, niveau m=E9thode de travail, je te conseille de modifier les sources=
 du
crawler directement, =E7a t'=E9vitera d'avoir =E0 les merger manuellement p=
ar la
suite. Au pire, si tu fais une bourde, les commandes "svn diff" et "svn
revert" sont l=E0 pour te filer un coup de main pour r=E9parer les choses. =
Et
tant que ton travail n'est pas commit=E9, tu ne risques pas de tout casser.
Ah, et si =E0 un moment tu veux qu'on jette un coup d'oeil au code, tu peux
envoyer directement ta sortie de "svn diff", on saura s'y retrouver. :-)

Je pense avoir fait le tour. Content que tu aies avanc=E9 en tout cas :-)

++

Guillaume




On Jan 10, 2008 4:50 PM, St=E9phan Gorget < ph...@gm...> wrote:

> Salut,
>
> j'ai peur que tu te sois tromper avec la fonction setUser, qui d'apr=E8s =
moi
> sert quand tu as une authentification (ex: htaccess). Dans notre cas il f=
aut
> rajouter un QHttpRequestHeader qu'il va falloir setter et ensuite passer =
en
> param=E8tre =E0 la fonction qui va initialiser la connection. J'ai trouv=
=E9 un
> vague exemple sur un forum (
> http://www.qtforum.de/forum/viewtopic.php?t=3D2255).
>
> St=E9phan
>
>
> On Jan 10, 2008 4:38 PM, Fabien Bourdoulous < fab...@gm...=
m>
> wrote:
>
> > Salut,
> > j'ai avanc=E9 un peu pour l'user-agent, mais je voudrais avoir
> > confirmation avant de l'inclure dans le code et faire les tests, si je =
ne
> > fait pas fausse route.
> > Donc j'en suis l=E0 :
> >
> >
> > 1. renseigner l'user-agent dans ICNetworkHandler : j'ai ajouter un
> > nouvel attribut (QString _userAgent) =E0 ICNetworkHandler. Mais peut =
=EAtre
> > ya-t-il une autre solution ? J'ai mis en "//TODO" les lignes =E0
> > ajouter/modifier :
> >
> >
> > /* ICRequestMaster */
> > _settings=3Dnew ICRequestSettings(parentseek);
> >
> > void ICRequestMaster::createRequest(const QUrl& url) {
> >   ICNetworkHandler* handler =3D ICNetworkHandlerFactory::create(url); /=
/
> > (original)
> >   //TODO: ICNetworkHandler* handler =3D
> > ICNetworkHandlerFactory::create(url, _settings->userAgent());
> > }
> >
> >
> > /* ICNetworkHandlerFactory : */
> > public: static ICNetworkHandler* create(QUrl url, QObject* parent=3DNUL=
L);
> > // (original)
> > //TODO: public: static ICNetworkHandler* create(QUrl url, QString agent=
,
> > QObject* parent=3DNULL);
> >
> > ICNetworkHandler* ICNetworkHandlerFactory::create(QUrl url, QObject*
> > parent) {
> > //TODO: ICNetworkHandler* ICNetworkHandlerFactory::create(QUrl url,
> > QString agent, QObject* parent) {
> >   ICNetworkHandler* retval =3D NULL;
> >   if(url.scheme() =3D=3D "http") {
> >     retval =3D new ICNetworkHttpHandler(parent);
> >     retval->setUrl(url);
> >     //TODO: retval->setUserAgent(agent);
> > }
> >
> >
> > /* ICNetworkHandler : */
> > //TODO: protected: QString _userAgent;
> > //TODO: public slots : virtual void setUserAgent(QString agent);
> >
> >
> >
> > 2. intialiser cette valeur dans ICNetworkHttpHandler : =E0 priori il n'=
y a
> > que l'userName de QHttp http =E0 modifier ?
> >
> > /* ICNetworkHttpHandler : */
> > QString ICNetworkHttpHandler::get()
> > {
> >   if (!processing()) {
> >     // (...)
> >     QHttp::ConnectionMode mode =3D url_.scheme().toLower() =3D=3D "http=
s" ?
> > QHttp::ConnectionModeHttps : QHttp::ConnectionModeHttp;
> >     http->setHost(url_.host(), mode, url_.port() =3D=3D -1 ? 0 :
> > url_.port());
> >     //TODO:http->setUser(_userAgent, NULL);
> > }
> >
> >
> > /* QHttp */
> > int QHttp::setUser ( const QString & userName, const QString & password
> > =3D QString() )
> > This function sets the user name userName and password password for web
> > pages that require authentication.
> >
> >
> >
> > 3. mettre firefox comme user-agent par d=E9faut ?
> >
> > // Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) Gecko/2006101=
0
> > Firefox/2.0
> > // Mozilla/5.0 (X11; U; Linux i686; fr; rv: 1.8.1.1) Gecko/20060601
> > Firefox/2.0.0.1 (Ubuntu-edgy)
> >
> > /* ICRequestSettings */
> > ICRequestSettings::ICRequestSettings(QObject*
> > parent):ICSettingsObject(parent){
> >   addSelfToCatalog();
> >
> >   _userAgent =3D "ICECrawler SVN"; // (original)
> >   //TODO: _userAgent =3D "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; =
rv:
> > 1.8.1) Gecko/20061010 Firefox/2.0"
> > }
> >
>
>

[Icecrawler-devel] Release & Meeting cancellation

[Guillaume C. <las...@gm...>]

Hey everyone.

First some highlights about our alpha1 release : It's not done yet. Truth is
I've been sick lately and quite frankly I've been unable to do any good when
trying to work, too.
I thought I was much better when getting up this morning, and also yesterday
at the Xmas show from Stravaganza, Chorut's and some others but given how
I'm feeling right now, I think I may have thought "victory" a little too
fast. That said, I prefer to cancel tonight's meeting and to get some more
rest instead.
I'm proposing a replacement for it, though : I'll be on IRC ( #icecrawler on
irc.freenode.net ) at the normal meeting time, so we can meet and talk
nevertheless. About the release, I'll try to get the job done tonight,
though I can't make any guarantee since it will largely depend on how I will
feel.

Thank you for your Comprehension

Guillaume

Re: [Icecrawler-devel] SVN problem

[Guillaume C. <las...@gm...>]

Since your behind our school proxy I think this may help you :
http://gcroes.com/blog/index.php?post/2007/11/13/SVN-Proxy

For ICECrawler it gives something like :


[groups]
sfnet=3D*.svn.sourceforge.net

(...)

[sfnet]
http-proxy-host=3D sigma.utc.fr
http-proxy-port=3D3128
http-proxy-username=3D
http-proxy-password=3D

Have a nice day !

Guillaume

On Nov 29, 2007 8:20 AM, Aurelien Beltrame <aur...@et...>
wrote:

>
> Hello !
>
> I have a little problem, I can't download source code with subversion.
> When I
> use it, it writes "it's impossible to connect to server"...
> I use ubuntu 7.10.
>
> Thanks for your advice
> Aur=E9lien
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by: The Future of Linux Business White Paper
> from Novell.  From the desktop to the data center, Linux is going
> mainstream.  Let it simplify your IT future.
> http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
> _______________________________________________
> Icecrawler-devel mailing list
> Ice...@li...
> https://lists.sourceforge.net/lists/listinfo/icecrawler-devel
>

[Icecrawler-devel] SVN problem

[Aurelien B. <aur...@et...>]

Hello !

I have a little problem, I can't download source code with subversion. When I
use it, it writes "it's impossible to connect to server"...
I use ubuntu 7.10.

Thanks for your advice
Aurélien

Re: [Icecrawler-devel] Absence

[Guillaume C. <las...@gm...>]

Hi :-)

Glad that you finally managed to get your computer working :) I just wanted
to drop a little note about your contribution to the project since you won't
be able to make it for tonight meeting. Could you please keep us informed
about what you're working on on the wiki or something ? And I think it would
be pretty cool to ask any question you may have to this list too, should you
have any. (too many questions are far better than having some unanswered,
and believe me, there are no dumb questions when you want to contribute
code)
I will discuss svn commit policies tonight but if you're working on
something and not really sure about it (which would be quite understandable
for a first-time code contributer), can you submit patches to this list so
we can review them ? (The "diff -u" and "svn diff" commands are your
friends).

We can also arrange an IRC meeting if you wish to.

See you later !

Guillaume

On Nov 28, 2007 10:45 AM, Pierre-Jean Baraud <pjb...@gm...> wrote:

> Hi men!
>
> I want to apologize for my absence this evening, but i won't be able
> to come to the meeting. Nevertheless, i have a good news (almost as
> good as the fact that our crawler crawls): my PC works!!
> I am working on ice crawler right now, i hope to be able to implement
> something (words scoring for instance) this week.
> Huge thanks to all devs who have worked hard to make IceCrawler crawl.
> see you soon,
>
> Pierre-Jean
>
> -------------------------------------------------------------------------
> SF.Net email is sponsored by: The Future of Linux Business White Paper
> from Novell.  From the desktop to the data center, Linux is going
> mainstream.  Let it simplify your IT future.
> http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
> _______________________________________________
> Icecrawler-devel mailing list
> Ice...@li...
> https://lists.sourceforge.net/lists/listinfo/icecrawler-devel
>

[Icecrawler-devel] Absence

[Pierre-Jean B. <pjb...@gm...>]

Hi men!

I want to apologize for my absence this evening, but i won't be able
to come to the meeting. Nevertheless, i have a good news (almost as
good as the fact that our crawler crawls): my PC works!!
I am working on ice crawler right now, i hope to be able to implement
something (words scoring for instance) this week.
Huge thanks to all devs who have worked hard to make IceCrawler crawl.
see you soon,

Pierre-Jean

[Icecrawler-devel] This week's good news :-)

[Guillaume C. <las...@gm...>]

Hi everyone !

I got kind of an announcement to make : Ready ?

Our crawler finally crawls !

After nearly seven months of development (has it really been this long
already ?) we finally got something that works... Almost. It's still needing
some key parts like the "Website" notion and at least a working export
module to actually do something useful. But still, this is great news as it
can now navigate on the web and follow the links it finds !
Really, it is news from the weekend actually (the commits are dated from
saturday and sunday), but since I partied a little bit all over friday and
saturday night, I figured out It would be better for me as well as for you
to wait one day before sharing it :-)

Congrats to everyone who made that possible (yeah, I'm also thinking of you,
lil' mexican girl ;-) ) and also thank you for the interest, ideas and faith
you've had for or in the project. (this time thinking of you, ICECrawler
newbies !)

I'd like also to take it as an opportunity to drive the project towards more
intensive development (at least on my side). I mean : c'mon guys ! This
thing actually does something ! Wouldn't it be great if it could reach its
full potential ? Speaking of which, our bugtracker got some items if you're
interested : http://sourceforge.net/tracker/?group_id=193794 :-) Do not
hesitate to fill it some more or to do some work on the items already listed
! (but before you do so, please send a mail to this list to check if someone
isn't already doing some work on what you plan to help us on)

Well, I gotta get back to work, I'm supposed to be in an internship :-p

See you later !

Guillaume

[Icecrawler-devel] Algo GEM

[Aurelien B. <aur...@et...>]

Bonjour,

ci-joint l'algorithme GEM.

Aurélien

//

Hello,

i send you GEM algorithm.

Aurelien

[Icecrawler-devel] Networking works

[Guillaume C. <las...@gm...>]

Mwhahahaha. At last !

Like always, almost, the bug was because of a tiny little thing... Well, for
those of you who care about knowing what was going wrong, here is a little
explanation :
The issue we had with the networking code was simple. Requests got done, but
we couldn't get any data from them. The tests were conducted using the
simple and well known URL "http://www.google.fr". Request any data out of
it, and you'll be given nothing... Almost. And this is wanted behaviour. Try
this little thing in a console (windows / linux / something else... It
should work on any platform) :
do a "telnet www.google.fr 80". Once it reports you are connected, type "GET
/ HTTP/1.0" followed by 2 strikes on your <return> / <enter> key. And look
at what you got :-) A redirect instruction for the user-agent (ICECrawler in
our case). We don't handle them for now and this is why we got no data. I
changed the test URL to "http://www.google.fr/index.html" and guess what ?
It worked !

I'll keep on working on the network module until I can get redirection
working along with some other things. My work should be committed and moved
to the main source tree tonight, moving the Network module from the temp
directory for good !

Have a nice end-of-the day !

Guillaume

[Icecrawler-devel] IC05 report

[Guillaume C. <las...@gm...>]

Just to let you know Franck got our report.

Guillaume

Re: [Icecrawler-devel] ICECrawler and the near future

[Guillaume C. <las...@gm...>]

Errr... Please forgive me for my English language mistakes. The more parts
of what I wrote I read, the more mistakes I can find. :-(

Note to self : posting at 1:40 AM is not always a good idea...

On 7/4/07, Guillaume Ceccarelli <las...@gm...> wrote:
>
> Hi everyone !
>
> I hope the holidays are going fine for all of you !
>
> Well... There are several things I'd like to talk about. Namely :
> - Staff
> - What misses to make ICECrawler work
> - Communication
> - Our brand new admin
> - Documentation for the project
> - The IC05 report
>
>
> === Staff ===
>
> Sooooo... Let's begin with the beginning : you ! Us ! The team !
> ICECrawler devels !
> To be able to work efficiently, I'd like to know when you guys will want
> to spend some time on the project this summer. No need to be really precise,
> but having a rough idea would be cool... For my part I should become active
> in the end of the week / start of the next week.
>
> === What misses to make ICECrawler work ===
>
> (As a tiny reminder)
>
> As you already know it, though almost all modules should work
> independently, ICECrawler a a whole doesn't work in its current state. The
> main issues now are :
> - The network module make requests but don't store its contents (which is
> a bug, not some sort of unimplemented thingy)
> - The GUI is a stub. We got ideas, most of us (at least I think) find
> these ideas cool but nevertheless, nothing in the UI is connected with the
> rest of the app, and there are still choices to make concerning the layout
> of our "crawl status" and "crawl settings" tabs
>
> When these issues will be addressed, I think we'll have a fair amount of
> testing to do too...
>
>
> === Communication ===
>
> Within our last semester we had the opportunity to meet once a week, which
> probably won't be the case for a while now. So to work together we'll have
> to adopt some means of communication. Looking at our ready-to-use tools, we
> have : the icecrawler mailing-lists, #icecrawler IRC channel, the ICECrawler
> wiki, Instant messaging using MSN / GTalk / Something else. I think the
> mailing-lists and IRC are the most effective way we have to talk about
> ICECrawler, but with one backside. Even if we are a team of full native
> French speakers now, this won't necessarily be the case in the future (be it
> near or distant). Beside that, the mailing-lists are also some kind of PR
> (Public Relations) thing. Since they are public, (excepted
> icecrawler-internal, to which almost no-one is subscribed now) they also act
> as some kind of advertising for our project. Someone interested in what we
> do (from a developer point of view) is likely to browse the mailing-list
> archives to take the temperature and see what is going on. And for that
> reasons, I'd really like to keep our mailing-lists English-speaking ones.
> That particular thing could be a drawback for some of us maybe... So I
> insist for you to tell me if it is the case for you, since the mailing-lists
> haven't been used by anyone excepted me so far. I also want to insist on the
> fact these lists are not only for me to post information. It should be
> something for everyone to use (even external people after all, since
> everyone can subscribe) to simply discuss about the project, ask questions,
> tell us what you're doing, and so on. Therefore please don't hesitate to
> post on it.
>
> Here are our set-up lists :
> - icecrawler-devel, the main development mailing list, which should be
> used almost every time we want to discuss something about the project
> - icecrawler-internal, a private mailing-list for which archives are not
> accessible to the public, which should be used to talk about confidential
> issues when there will be some (like security issues / things like that)
> - icecrawler-announce, which will be used when we will be more mainstream
> to announce new versions, things like that
> - icecrawler-commits, a brand new one, to which a message is sent by sf.net
> svn server whenever a commit occurs, allowing us to keep tracks of them.
>
> Oh, and since it is communication-related too, the developer section from
> the wiki is now readable by anyone (it's still only editable by us though)
>
> One other thing, starting now, i'll be on our IRC channel whenever I'm
> online so you can find me there if you want to.
>
> === Our brand new admin ===
>
> I'd like to introduce you to phantez :-) , who is now an admin for the
> project at SF.net and the wiki. So if something goes wrong or if some
> maintenance is needed for any reason, we are at least two people to be able
> to provide the service. It should help us to avoid problems if one of us
> would be unavailable at some time.
>
>
> === Documentation for the project ===
>
> One thing that must be completed at least when ICECrawler will be up and
> working, would be to fully document our source code using Doxygen or
> something else is someone has a better alternative to tell us about. For my
> part I still have the UML diagrams made with Visual Paradigm without having
> them finished nor published. I will make sure this is corrected more or less
> quickly
>
>
> === The IC05 report ===
>
> We may be having a problem here... I assume you all received my e-mail
> with the report attached for Franck to reviewed, I asked him to reply when
> he'd have received, but he haven't yet. I wrote him again today. I don't
> know if he has answered to someone else or something, so please tell me if
> you know something I don't about this...
>
>
> I think that's all. I'm so sorry I couldn't make this e-mail shorter.
> Still, have a nice day !
>
> Guillaume
>

[Icecrawler-devel] ICECrawler and the near future

[Guillaume C. <las...@gm...>]

Hi everyone !

I hope the holidays are going fine for all of you !

Well... There are several things I'd like to talk about. Namely :
- Staff
- What misses to make ICECrawler work
- Communication
- Our brand new admin
- Documentation for the project
- The IC05 report


=== Staff ===

Sooooo... Let's begin with the beginning : you ! Us ! The team ! ICECrawler
devels !
To be able to work efficiently, I'd like to know when you guys will want to
spend some time on the project this summer. No need to be really precise,
but having a rough idea would be cool... For my part I should become active
in the end of the week / start of the next week.

=== What misses to make ICECrawler work ===

(As a tiny reminder)

As you already know it, though almost all modules should work independently,
ICECrawler a a whole doesn't work in its current state. The main issues now
are :
- The network module make requests but don't store its contents (which is a
bug, not some sort of unimplemented thingy)
- The GUI is a stub. We got ideas, most of us (at least I think) find these
ideas cool but nevertheless, nothing in the UI is connected with the rest of
the app, and there are still choices to make concerning the layout of our
"crawl status" and "crawl settings" tabs

When these issues will be addressed, I think we'll have a fair amount of
testing to do too...


=== Communication ===

Within our last semester we had the opportunity to meet once a week, which
probably won't be the case for a while now. So to work together we'll have
to adopt some means of communication. Looking at our ready-to-use tools, we
have : the icecrawler mailing-lists, #icecrawler IRC channel, the ICECrawler
wiki, Instant messaging using MSN / GTalk / Something else. I think the
mailing-lists and IRC are the most effective way we have to talk about
ICECrawler, but with one backside. Even if we are a team of full native
French speakers now, this won't necessarily be the case in the future (be it
near or distant). Beside that, the mailing-lists are also some kind of PR
(Public Relations) thing. Since they are public, (excepted
icecrawler-internal, to which almost no-one is subscribed now) they also act
as some kind of advertising for our project. Someone interested in what we
do (from a developer point of view) is likely to browse the mailing-list
archives to take the temperature and see what is going on. And for that
reasons, I'd really like to keep our mailing-lists English-speaking ones.
That particular thing could be a drawback for some of us maybe... So I
insist for you to tell me if it is the case for you, since the mailing-lists
haven't been used by anyone excepted me so far. I also want to insist on the
fact these lists are not only for me to post information. It should be
something for everyone to use (even external people after all, since
everyone can subscribe) to simply discuss about the project, ask questions,
tell us what you're doing, and so on. Therefore please don't hesitate to
post on it.

Here are our set-up lists :
- icecrawler-devel, the main development mailing list, which should be used
almost every time we want to discuss something about the project
- icecrawler-internal, a private mailing-list for which archives are not
accessible to the public, which should be used to talk about confidential
issues when there will be some (like security issues / things like that)
- icecrawler-announce, which will be used when we will be more mainstream to
announce new versions, things like that
- icecrawler-commits, a brand new one, to which a message is sent by
sf.netsvn server whenever a commit occurs, allowing us to keep tracks
of them.

Oh, and since it is communication-related too, the developer section from
the wiki is now readable by anyone (it's still only editable by us though)

One other thing, starting now, i'll be on our IRC channel whenever I'm
online so you can find me there if you want to.

=== Our brand new admin ===

I'd like to introduce you to phantez :-) , who is now an admin for the
project at SF.net and the wiki. So if something goes wrong or if some
maintenance is needed for any reason, we are at least two people to be able
to provide the service. It should help us to avoid problems if one of us
would be unavailable at some time.


=== Documentation for the project ===

One thing that must be completed at least when ICECrawler will be up and
working, would be to fully document our source code using Doxygen or
something else is someone has a better alternative to tell us about. For my
part I still have the UML diagrams made with Visual Paradigm without having
them finished nor published. I will make sure this is corrected more or less
quickly


=== The IC05 report ===

We may be having a problem here... I assume you all received my e-mail with
the report attached for Franck to reviewed, I asked him to reply when he'd
have received, but he haven't yet. I wrote him again today. I don't know if
he has answered to someone else or something, so please tell me if you know
something I don't about this...


I think that's all. I'm so sorry I couldn't make this e-mail shorter. Still,
have a nice day !

Guillaume

[Icecrawler-devel] First message :-)

[Guillaume C. <las...@gm...>]

Hello everyone !

Here is the very first message for our project development mailing-list.
Hurray !

As it is a bit long, here is a short summary :

- About this list
- News !
  - Project wiki
  - #icecrawler IRC Channel
  - Next ICECrawler meeting

==== About this list ====

This list is now live ! :-)
Now we can discuss anything concerning the development of ICECrawler on
here. Just send an e-mail to ice...@li... and
you're done !

My use of the English language is intentional. Should anyone have concerns
with using English in the mailing-list, please let me know. A mailing-list
is a communication tool, and therefore it shouldn't bring communication
issues between the members of our team. Since this particular list will be a
main communication place for the development of ICECrawler, everyone in here
should be able to understand and post on it without difficulties. That being
said, if the use of the English language doesn't raise any concern for
anyone of us, I think this list should remain English-only, mainly for these
reasons :
- Communication issues : This is a public mailing-list, which means
absolutely anyone can subscribe to it or read its archives. Should anyone be
interested in our project, this person should be able to communicate with us
without the fear of not being understood or of not being in the right place
to do so.
- Public Relations : As you know, we are hosted on SourceForge. SF.net is
the leader in open-source project hosting. As such, many people browse
sf.net project listings to look for projects in specific categories. Our
project could be discovered by some people in this way and become interested
in it. (see previous point)
- Durability : At that time, no one knows how this project will evolve. The
same goes for the development team. Perhaps you won't want to carry on
working on ICECrawler after the end of this semester. Likewise, even if I do
wish to continue working on it, I could not be able to do so anymore at some
point. Having the main development list in English is a sort of guarantee
this project won't die, should we ever give it up.
- English language experience : I don't think many of us use English as a
daily language. Here is our chance ! :-) There is no better way to learn or
to improve one's skills for a language than to actually use it in "real"
situations, and on a regular basis. We don't need to be English-language
gurus by the way. What's the most important is to be understood, and I think
we can be pretty well !


==== News ! ====

Phewww... The long introduction is now over :-). Now let's welcome the real
news !


=== Project Wiki ===

The ICECrawler wiki is online at icecrawler.sourceforge.net. It should be
our main "blackboard" for anything related to the project. It is both a
communication and a work tool. Most documentation about what we do and how
we do it should go there. I made the decision to make the development area
private until the official end of IC05. After all, we're in a competition
right now ! ;-) Please go there and create an account so I'll give it some
rights. By the way, we're in a tough period right now because of our exams,
and so am I. I don't think I will upload the current status of my work to
the wiki until Tuesday.


=== #icecrawler IRC Channel ===

I've set up an IRC channel : #icecrawler on irc.sourceforge.net. I'll do my
best to be online as much as possible. This channel should be a way for us
to simply have a chat about the project (or anything else :-) ) or to
discuss anything. As we probably won't be seeing a lot one another outside
of IC05 classes, I thought it could be a way to talk without having to meet
physically.


=== Next ICECrawler meeting ===

Now a real question : Tuesday is normally our day for IC05 and thus our day
for ICECrawler, too. As you know, the next Tuesday is a holiday for us. I'm
OK for a meeting on this day, but only if we actually have something to say
or do together. Since I think we are all working for our exams, it won't
necessarily be the case. For my part, I'll be working on ICECrawler on the
next days. If you'll have begun to work on something before Tuesday in the
evening and wish us to meet, please tell us by responding to this list. The
same goes if you think a meeting on Tuesday is useless. Note however that a
meeting could be a meeting on IRC, too.


That's all folks !
Seeya and have a nice sunday !


Guillaume