[Seeks-users] wrapup of meeting on Seeks' connectors and parsers

[Emmanuel B. <ebe...@se...>]

Hi all,

below is a wrapup of our Seeks' most recent meeting of last March
2nd and 3rd, on IRC.

The discussion mostly involved Stephane Bonhomme (waloo), Camille
Harang (mammique) and myself (beniz), with interventions from Pablo
Joubert (piti), on parsers, scrapers, and input connectors for Seeks'.

As a reminder, Seeks acts as a decentralized collaborative filter on
top of an arbitrary number of personal and public feeds. Typically,
most common feeds include results from conventional search engines,
thus building a meta-search engine.

Logs from the discussion are available here:
http://redmine.seeks-project.info/issues/336

The goal of the discussion was to:
- assess the parser theoretical complexity and technical features;
- discuss how to make parsers more expressive ane easier at the same time. 
This would benefit the project twice: by lowering the
difficulty to contribute to Seeks; by opening the way to a jungle of
tiny parsers / scrapers, able to cover a large part of the Web (an
idea defended by mammique extensively in an earlier thread).

First we reviewed the two types of parsers, tree-based and
event-based. The latter are of linear complexity while tree-based
exhibit a log-linear complexity. For this reason, until now, Seeks has
only implemented event-based parsers in the form of SAX
parsers. However, SAX parsers are not robust to tiny changes in the
feed structure, and are expensive to tweak.
Tree-based parsers instead allow to express powerful queries, and thus
reduce a parser to declarative expressions in a given
language. Typically, xpath was recommended by waloo as the number one
way to go for this type of parsers. Tree-based is more robust, and
fits with all dominant dom-like data structures on the Web (rss / atom 
/ html / xml).
Example, it allows to say a 'snippet' is /div[3]/span[12]/after:h2
Waloo sums it up saying 'the balance is between the efficiency / and
power of expression'.

Next was mentioned the novel ability to check a grammar with libxml2,
while parsing it. I'm not sure this has to be considered right now.
This seems a path to explore though.

On the application side, xpath may not allow to post-process the
scrapped snippets (output of the parsers), while a SAX parser does it
within its treatment loop (since this loop is directly written in C++
in our case, as opposed to the declarative query language of xpath).

Next, it was agreed that we should retain the SAX parsing capability
while adding support for at least xpath. Further discussion the next
day led to the inclusion of more declarative formats. Typically,
support for CSS selectors, regexp (existing), and JSON (input) should
come at some point later. xpath remains the first format to add
support for.

The idea remains to have a number of connectors, at least one per
feed source type. The connectors would be instanciated at
startup (or hot-pluggable) by reading either a declarative description
or by using a derived SAX parser.

Waloo sketches a declarative parser description roughly as:
- a parse name (unique);
- a xpath expression identifying the list of snippets;
- an relative xpath expression for each snippet element you need;

Finally, it was agreed that OpenSearch response element
(http://www.opensearch.org/Specifications/OpenSearch/1.1#OpenSearch_response_elements)
should be supported ASAP with a SAX parser.
This is ticket http://redmine.seeks-project.info/issues/226

From there, the current path seems to be:
a/ write a parser configuration file as proposed by waloo. This file
would contain what elements are crucially needed to be parsed from
input put feed (e.g. title, URL, summary, ...), and some optional
fields.
b/ Study, design & implement a basic architecture for connectors
(parsers, scrapers) better / simpler / more flexible integration in
the 'websearch' plugin.

Waloo has started working on a/. (I'm missing the URL right now).

I should be able to take care of b/ after I'm done with my current bag
of tickets. Unless someone else jumps in. In a short future, I'll put
down the chain of tickets on the redmine for completing b/.

Waloo, Mammique, thanks for taking the time of thinking and then
discussing these issues. These extensions of Seeks to the outside
world should prove key in the future.

Em.