connotea-code-devel — Discussion and devlopment of the core Connotea code and plug-ins.

[Connotea-code-devel] Connotea on CPAN

[Mulvany, I. <i.m...@na...>]

Hi Kieren,

Great idea. I've just applied for a CPAN account, and I will upload the code
there when we get the account.

- Ian

Subject: Re: [Connotea-code-devel] perl api -- modifications

Can you push a version to CPAN as well to improve the visibility of
the connotea project please.  Tell me to jfdi and i'll push a copy to
cpan using my username (ZARQUON fwiw).

Kieren

On 13/10/2007, Martin Flack <ma...@ne...> wrote:
> Hi Philip,
>
> Thanks for this. I think the Java and Ruby libraries have been getting
> more attention so we must have missed these issues. I've prepared a new
> tarball labeled 0.1.1 which will go up on Sourceforge soon.
>
> Martin
>
> Philip Bett wrote:
> > Hi all,
> > I've had to make a couple of modifications to my copy of Connotea.pm in
> > www-connotea-perl-0.1 in order to get it to work -- I thought I'd post
> > them here in case others found them useful. (apologies if they're known
> > issues, but they are still in the downloadable code)
> >
> >
> > (1) Using a proxy:
> > Working from a university means all web traffic has to go via a proxy
> > server.
> >
> > I had to add a line to the sub 'new' in Connotea.pm:
> > ...
> > $ua = LWP::UserAgent->new;                   ## Existing line...
> > $ua->env_proxy;                              ## New line!
> > $ua->agent(UA_PREFIX.' ('.$ua->agent.')');   ## Existing line...
> > ...
> >
> > Simple enough.
> >
> >
> > (2) Parsing the response:
> > This is a bit more serious.  The sub 'parse_body' is set to croak if the
> > returned content-type isn't on a list of allowed values.  This list
> > includes "application/xml", but the actual response gives a content-type
> > of "application/xml; charset=UTF-8", so it will always croak.
> >
> > My solution is to split the returned string at the first semicolon, and
> > just take the first chunk:
> >
> > @conttypes = split( /([;])/, $response->header('Content-type'), );
> > $conttype  = $conttypes[0];
> > croak "Invalid content type in response: $conttype\n" unless
> > $rdf_content_types{$conttype};
> >
> > (original line was:
> > croak 'Invalid content type in response:
> > "'.$response->header('Content-type').'"' unless
> > $rdf_content_types{$response->header('Content-type')};
> > )
> >
> >
> > These solutions are, I'm sure, not the most efficient, elegant or
> > robust, but that's about the limit of my perl skills at the moment!  I'd
> > like to hear how these could be improved, or if anyone knows of any
> > similar pitfalls I might encounter.
> >
> >
> > Cheers,
> > Phil Bett
> >
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems?  Stop.
> > Now Search log events and configuration files using AJAX and a browser.
> > Download your FREE copy of Splunk now >> http://get.splunk.com/
> > _______________________________________________
> > Connotea-code-devel mailing list
> > Con...@li...
> > https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Connotea-code-devel mailing list
> Con...@li...
> https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
>

********************************************************************************   
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan 
Publishers Limited or its agents by means of e-mail communication. Macmillan 
Publishers Limited Registered in England and Wales with registered number 785998 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************

Re: [Connotea-code-devel] perl api -- modifications

[Kieren D. <di...@gm...>]

Can you push a version to CPAN as well to improve the visibility of
the connotea project please.  Tell me to jfdi and i'll push a copy to
cpan using my username (ZARQUON fwiw).

Kieren

On 13/10/2007, Martin Flack <ma...@ne...> wrote:
> Hi Philip,
>
> Thanks for this. I think the Java and Ruby libraries have been getting
> more attention so we must have missed these issues. I've prepared a new
> tarball labeled 0.1.1 which will go up on Sourceforge soon.
>
> Martin
>
> Philip Bett wrote:
> > Hi all,
> > I've had to make a couple of modifications to my copy of Connotea.pm in
> > www-connotea-perl-0.1 in order to get it to work -- I thought I'd post
> > them here in case others found them useful. (apologies if they're known
> > issues, but they are still in the downloadable code)
> >
> >
> > (1) Using a proxy:
> > Working from a university means all web traffic has to go via a proxy
> > server.
> >
> > I had to add a line to the sub 'new' in Connotea.pm:
> > ...
> > $ua = LWP::UserAgent->new;                   ## Existing line...
> > $ua->env_proxy;                              ## New line!
> > $ua->agent(UA_PREFIX.' ('.$ua->agent.')');   ## Existing line...
> > ...
> >
> > Simple enough.
> >
> >
> > (2) Parsing the response:
> > This is a bit more serious.  The sub 'parse_body' is set to croak if the
> > returned content-type isn't on a list of allowed values.  This list
> > includes "application/xml", but the actual response gives a content-type
> > of "application/xml; charset=UTF-8", so it will always croak.
> >
> > My solution is to split the returned string at the first semicolon, and
> > just take the first chunk:
> >
> > @conttypes = split( /([;])/, $response->header('Content-type'), );
> > $conttype  = $conttypes[0];
> > croak "Invalid content type in response: $conttype\n" unless
> > $rdf_content_types{$conttype};
> >
> > (original line was:
> > croak 'Invalid content type in response:
> > "'.$response->header('Content-type').'"' unless
> > $rdf_content_types{$response->header('Content-type')};
> > )
> >
> >
> > These solutions are, I'm sure, not the most efficient, elegant or
> > robust, but that's about the limit of my perl skills at the moment!  I'd
> > like to hear how these could be improved, or if anyone knows of any
> > similar pitfalls I might encounter.
> >
> >
> > Cheers,
> > Phil Bett
> >
> >
> > -------------------------------------------------------------------------
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems?  Stop.
> > Now Search log events and configuration files using AJAX and a browser.
> > Download your FREE copy of Splunk now >> http://get.splunk.com/
> > _______________________________________________
> > Connotea-code-devel mailing list
> > Con...@li...
> > https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Connotea-code-devel mailing list
> Con...@li...
> https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
>

Re: [Connotea-code-devel] perl api -- modifications

[Martin F. <ma...@ne...>]

Hi Philip,

Thanks for this. I think the Java and Ruby libraries have been getting 
more attention so we must have missed these issues. I've prepared a new 
tarball labeled 0.1.1 which will go up on Sourceforge soon.

Martin

Philip Bett wrote:
> Hi all,
> I've had to make a couple of modifications to my copy of Connotea.pm in 
> www-connotea-perl-0.1 in order to get it to work -- I thought I'd post 
> them here in case others found them useful. (apologies if they're known 
> issues, but they are still in the downloadable code)
> 
> 
> (1) Using a proxy:
> Working from a university means all web traffic has to go via a proxy 
> server.
> 
> I had to add a line to the sub 'new' in Connotea.pm:
> ...
> $ua = LWP::UserAgent->new;                   ## Existing line...
> $ua->env_proxy;                              ## New line!
> $ua->agent(UA_PREFIX.' ('.$ua->agent.')');   ## Existing line...
> ...
> 
> Simple enough.
> 
> 
> (2) Parsing the response:
> This is a bit more serious.  The sub 'parse_body' is set to croak if the 
> returned content-type isn't on a list of allowed values.  This list 
> includes "application/xml", but the actual response gives a content-type 
> of "application/xml; charset=UTF-8", so it will always croak.
> 
> My solution is to split the returned string at the first semicolon, and 
> just take the first chunk:
> 
> @conttypes = split( /([;])/, $response->header('Content-type'), );
> $conttype  = $conttypes[0];
> croak "Invalid content type in response: $conttype\n" unless 
> $rdf_content_types{$conttype};
> 
> (original line was:
> croak 'Invalid content type in response: 
> "'.$response->header('Content-type').'"' unless 
> $rdf_content_types{$response->header('Content-type')};
> )
> 
> 
> These solutions are, I'm sure, not the most efficient, elegant or 
> robust, but that's about the limit of my perl skills at the moment!  I'd 
> like to hear how these could be improved, or if anyone knows of any 
> similar pitfalls I might encounter.
> 
> 
> Cheers,
> Phil Bett
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> Connotea-code-devel mailing list
> Con...@li...
> https://lists.sourceforge.net/lists/listinfo/connotea-code-devel

[Connotea-code-devel] perl api -- modifications

[Philip B. <p.e...@du...>]

Hi all,
I've had to make a couple of modifications to my copy of Connotea.pm in 
www-connotea-perl-0.1 in order to get it to work -- I thought I'd post 
them here in case others found them useful. (apologies if they're known 
issues, but they are still in the downloadable code)


(1) Using a proxy:
Working from a university means all web traffic has to go via a proxy 
server.

I had to add a line to the sub 'new' in Connotea.pm:
...
$ua = LWP::UserAgent->new;                   ## Existing line...
$ua->env_proxy;                              ## New line!
$ua->agent(UA_PREFIX.' ('.$ua->agent.')');   ## Existing line...
...

Simple enough.


(2) Parsing the response:
This is a bit more serious.  The sub 'parse_body' is set to croak if the 
returned content-type isn't on a list of allowed values.  This list 
includes "application/xml", but the actual response gives a content-type 
of "application/xml; charset=UTF-8", so it will always croak.

My solution is to split the returned string at the first semicolon, and 
just take the first chunk:

@conttypes = split( /([;])/, $response->header('Content-type'), );
$conttype  = $conttypes[0];
croak "Invalid content type in response: $conttype\n" unless 
$rdf_content_types{$conttype};

(original line was:
croak 'Invalid content type in response: 
"'.$response->header('Content-type').'"' unless 
$rdf_content_types{$response->header('Content-type')};
)


These solutions are, I'm sure, not the most efficient, elegant or 
robust, but that's about the limit of my perl skills at the moment!  I'd 
like to hear how these could be improved, or if anyone knows of any 
similar pitfalls I might encounter.


Cheers,
Phil Bett

Re: [Connotea-code-devel] api help - basic authentication

[Martin F. <ma...@ne...>]

Hi Arindam,

 > Now it seems like request.open only works on files on the localhost.
 > If that's true connotea will always return a 'permission denied to
 > call open()'.

This sounds like a security model restriction in your browser perhaps?

Martin

[Connotea-code-devel] api help - basic authentication

[Ganguly, A. <aga...@so...>]

Hello,

  I'm attempting to use the api to add and retrieve content. Saw the
javascript code under http://www.connotea.org/wiki/WebAPIAuthentication

=20

Basically replicated it to,

=20

request =3D window.XMLHttpRequest ? new XMLHttpRequest() : new
ActiveXObject("MSXML2.XMLHTTP.3.0");

if (!request)

  alert("Error initializing XMLHttpRequest!");

           =20

var url =3D "http://www.connotea.org/data/noop";

           =20

request.open("GET", url, true); --------------------------fails here

request.setRequestHeader("Authorization",
"Basic"+btoa(login+":"+password));

request.onreadystatechange =3D handleResponse;

request.send(null);

=20

handle Response is this,

=20

if (request.readyState =3D=3D 4)

       if (request.status =3D=3D 200)

         alert("Server is done!");

       else if (request.status =3D=3D 404)

         alert("Request URL does not exist");

       else

         alert("Error: status code is " + request.status);=20

=20

=20

Now it seems like request.open only works on files on the localhost. If
that's true connotea will always return a 'permission denied to call
open()'.

=20

Is there a way around this? The goal is to do any authentication/
response handling in the background w/o showing any of the ugly response
headers.

=20

I expect to run into a similar issue on trying to do a post request
using HttpRequest.

=20

Can someone help please.

=20

Arindam Ganguly

Calit2, UCSD

=20

=20

Re: [Connotea-code-devel] Connotea-code-devel Digest, Vol 17, Issue 2

[Mulvany, I. <i.m...@na...>]

Hi Andrew,

Not at the moment, but we want to provide this functionality later this
year.

- Ian

On 18/9/07 20:15, "con...@li..."
<con...@li...> wrote:

> Send Connotea-code-devel mailing list submissions to
> con...@li...
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
> or, via email, send a message with subject or body 'help' to
> con...@li...
> 
> You can reach the person managing the list at
> con...@li...
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Connotea-code-devel digest..."
> 
> 
> Today's Topics:
> 
>    1. get connotea bookmarklet IDs from PubMed ID? (Andrew Straw)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Mon, 17 Sep 2007 13:03:07 -0700
> From: Andrew Straw <str...@as...>
> Subject: [Connotea-code-devel] get connotea bookmarklet IDs from
> PubMed ID?
> To: con...@li...
> Message-ID: <46E...@as...>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> Hi,
> 
> Is there a way to query the Connotea web services API if there are any
> bookmarklets in the Connotea DB based on the PubMed ID?
> 
> Thanks,
> Andrew
> 
> 
> 
> ------------------------------
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> 
> ------------------------------
> 
> _______________________________________________
> Connotea-code-devel mailing list
> Con...@li...
> https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
> 
> 
> End of Connotea-code-devel Digest, Vol 17, Issue 2
> **************************************************


-- 
Ian Mulvany
i.m...@na...
+44(0)2078434756
www.nature.com
www.connotea.org
--



********************************************************************************   
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan 
Publishers Limited or its agents by means of e-mail communication. Macmillan 
Publishers Limited Registered in England and Wales with registered number 785998 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************

[Connotea-code-devel] get connotea bookmarklet IDs from PubMed ID?

[Andrew S. <str...@as...>]

Hi,

Is there a way to query the Connotea web services API if there are any
bookmarklets in the Connotea DB based on the PubMed ID?

Thanks,
Andrew

Re: [Connotea-code-devel] New connotea code release

[Adie, E. <e....@na...>]

> o Alpha-version stand-alone citation server capability.

If I can big up this bullet point a bit: the idea behind this is that you
can use the Connotea code as a web service - POST an URI and get back the
citation for whatever resource it points to in JSON, XML or RIS formats.

Should hopefully make literature based mashups much, much easier: no more
regexp'ing out DOIs in Greasemonkey scripts...

Euan

********************************************************************************   
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan 
Publishers Limited or its agents by means of e-mail communication. Macmillan 
Publishers Limited Registered in England and Wales with registered number 785998 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************

[Connotea-code-devel] New connotea code release

[Mulvany, I. <i.m...@na...>]

We have soft-released the new codebase for conntoea. It has been over a year
since the last release of the stable version, and so there have a few
changes.

You can download the code from the sourceforge connotea page:
http://sourceforge.net/projects/connotea

This is a bets release because we have not tested the install on a clean
system yet, and when we do we may make a few changes, so you can possibly
expect an updated release in two weeks.

What=B9s changed? I=B9ve listed info from the release notes below, but one =
main
change is that we have re-arranged things so that the full 1.8 release will
be available via a darcs repository into which we envision people can
directly submit patches.

And on that note I=B9m off bouldering in France for a week.

    * Web API in regular use.
          o Template Toolkit based templates in regular use.
          o More optimized SQL queries for common requests.
          o Greater use of transactions in MySQL.
          o Greater flexbility for citation source modules.
          o New citation source modules, plus improvements to existing
modules.
          o Blog component to create news page from external blog.
          o Wiki component to create custom wiki.
          o Admin component with user search.
          o Integration with Bibutils library for BibTeX and MODS output.
          o Antispam system with captcha and quarantine responses.
          o Click tracker for all posts.
          o Alpha-version proxy module system to handle known proxied post
URL=B9s.
          o Alpha-version stand-alone citation server capability.
          o Additional tools such as command-line post by API, user
recovery, and test suite launch.
          o Automated deployment scripts, now supporting Darcs instead of
CVS.
          o Updated code to support newer versions of CPAN modules.
          o More test suite scripts.

--=20
Ian Mulvany
i=2Em...@na...
+44(0)2078434756
www.nature.com
www.connotea.org
--



***************************************************************************=
*****  =20
DISCLAIMER: This e-mail is confidential and should not be used by anyone wh=
o is
not the original intended recipient. If you have received this e-mail in er=
ror
please inform the sender and delete it from your mailbox or any other stora=
ge
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agen=
ts.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail =
or
its attachments and it is your responsibility to scan the e-mail and=20
attachments (if any). No contracts may be concluded on behalf of Macmillan=20
Publishers Limited or its agents by means of e-mail communication. Macmilla=
n=20
Publishers Limited Registered in England and Wales with registered number 7=
85998=20
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS  =20
***************************************************************************=
*****

[Connotea-code-devel] a new tool for tagging

[Benjamin G. <go...@in...>]

Dear Connoteaites,

  I'm writing to let you know about a new GreaseMonkey extension to  
Connotea that I hope you might be interested in trying out.  Very  
briefly, it (ED the Entity Describer) provides social taggers (you)  
with an easy way to utilize terms from existing controlled  
vocabularies (such as MeSH) in combination with your own tags.  It is  
designed to provide many of the benefits of controlled vocabularies  
(consistency, non-ambiguity, relations between terms) while  
maintaining the all of the benefits of tagging (speed, flexibility,  
personalization,).

  I hope that you find it useful and fun.

  You can find out more about what ED does, how to install it, and  
how to use it on its wiki page: http://www.connotea.org/wiki/ 
EntityDescriber

  Please do let me know what you think about it (its part of my PhD  
research).

best regards
-Ben

[Connotea-code-devel] FW: Connotea Java API Client

[Mulvany, I. <i.m...@na...>]

> From: "Townson, Chris" <C.T...@na...>

> 
> fyi: this is now on Google Code (under GPL) at
> http://code.google.com/p/connotea-java/
> 
> C 

********************************************************************************   
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan 
Publishers Limited or its agents by means of e-mail communication. Macmillan 
Publishers Limited Registered in England and Wales with registered number 785998 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************

Re: [Connotea-code-devel] connotea java api client

[Christopher T. <chr...@ch...>]

Christopher Townson wrote:
> Hi all,
> 
> I've packaged up an early beta of a Java client for the Connotea API.
> There are still a few outstanding "todos" (e.g. write a proper unit test
> for the main client class ;D) but it is basically ready and comments
> from anyone who wants to try it out now would be very welcome.
> 
> At present, there is only a pre-compiled jar and javadocs. The source
> etc will be made available as soon as we have found a permanent home and
> determined what licence the code should be released under etc etc.
> 
> Temporary homes for both of the above:
> 
> jar file: http://nascent.nature.com/connotea/connotea.jar
> api docs: http://nascent.nature.com/connotea/api/
> 
> Please do try it out and report back any bugs you encounter to this
> list, or directly to me (via author link in api docs).
> 
> Likewise, if there are any glaring omissions or misrepresentations of
> the Connotea API, just let me know.
> 
> Cheers,
> 
> Chris
> 

Hi all,

I have made a permanent home for the Connotea Java API client on Google
Code at http://code.google.com/p/connotea-java/

(Please note that the temporary home detailed in my previous message
will now be retired.)

Source and binary distribution packages can be downloaded from the
Google Code project homepage, alongside the usual bug tracking, wiki,
and svn facilities.

It is released under the GPL.

Contributions/feedback are most welcome.

I hope to progress the project over the coming months as time allows -
primarily in the form of a more robust and thorough test suite.

I hope this code proves useful and usable.

Cheers,

Chris

[Connotea-code-devel] Mailing lists under moderation

[Mulvany, I. <i.m...@na...>]

Our two mailing lists for connotea were experiencing a high load of spam so
I have turned on full moderation for all posts. If you are posting to the
lists then it might take a day before your post goes out, and if you post
over the weekend it will get queued until Monday. Hopefully this will reduce
the number of spam messages that you receive from our mailing lists.

And just a reminder, the lists are the "connotea-discuss
https://lists.sourceforge.net/lists/listinfo/connotea-discuss
 for general discussion

And 

 connotea-code-devel
https://lists.sourceforge.net/lists/listinfo/connotea-code-devel for
discussing the source code.


We also now have a blog for news at http://www.connotea.org/blog


    
-- 
Ian Mulvany
i.m...@na...
+44(0)2078434756
www.nature.com
www.connotea.org
--



********************************************************************************   
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan 
Publishers Limited or its agents by means of e-mail communication. Macmillan 
Publishers Limited Registered in England and Wales with registered number 785998 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************

Re: [Connotea-code-devel] connotea java api client

[Christopher T. <chr...@ch...>]

Christopher Townson wrote:
> Hi all,
> 
> I've packaged up an early beta of a Java client for the Connotea API.
> There are still a few outstanding "todos" (e.g. write a proper unit test
> for the main client class ;D) but it is basically ready and comments
> from anyone who wants to try it out now would be very welcome.
> 
> At present, there is only a pre-compiled jar and javadocs. The source
> etc will be made available as soon as we have found a permanent home and
> determined what licence the code should be released under etc etc.
> 
> Temporary homes for both of the above:
> 
> jar file: http://nascent.nature.com/connotea/connotea.jar
> api docs: http://nascent.nature.com/connotea/api/
> 
> Please do try it out and report back any bugs you encounter to this
> list, or directly to me (via author link in api docs).
> 
> Likewise, if there are any glaring omissions or misrepresentations of
> the Connotea API, just let me know.
> 
> Cheers,
> 
> Chris
> 

Hi all,

I have made a permanent home for the Connotea Java API client on Google 
Code at http://code.google.com/p/connotea-java/

(Please note that the temporary home detailed in my previous message 
will now be retired.)

Source and binary distribution packages can be downloaded from the 
Google Code project homepage, alongside the usual bug tracking, wiki, 
and svn facilities.

It is released under the GPL.

Contributions/feedback are most welcome.

I hope to progress the project over the coming months as time allows - 
primarily in the form of a more robust and thorough test suite.

I hope this code proves useful and usable.

Cheers,

Chris

Re: [Connotea-code-devel] more on DBIx::Class::CDBICompat

[Kieren D. <ki...@di...>]

On 14 Aug 2007, at 02:18, Martin Flack wrote:

> Kieren Diment wrote:
>> so this is an example of a problematic line:
>> my $time = $Bibliotech::Apache::QUICK{DB} ||   
>> $Bibliotech::Apache::MEMCACHE->get('DB');
>
> We fetch the latest database update time from the memcache (to  
> calculate last_updated times) and we memoize it so that we don't  
> keep asking memcache over and over in the same request.
>
> This is in service of being able to do proper 304 HTTP status  
> codes; we try not to hit the database if we know that the database  
> has not even been changed since the time the user reports they last  
> saw the page they are now requesting again. The various components  
> report a last_updated time and for the data-driven components its  
> usually just this DB time.
>
> This is by definition controller and model interacting.
>
> Do you know how this generally would be accomplished in Catalyst?


OK.  Throught the catalyst request cycle, you get access to a  
persistent $c object.  $c contains  things like $c->request and $c- 
 >response, $c->user (provided by the auth plugins) $c->action and so  
on.  In general the $c object is available in everything that does  
"use base 'Catalyst::Controller'" and use "base 'Catalyst::View'".   
Models are slightly different and by default you don't get $c -  
they're just a package automatically picked up by  
Module:::Pluggable::Object when the app runs initially.  However if  
your model supports the ACCEPT_CONTEXT method (documented in  
Catalyst::Manual::Intro) then you do get access to $c in your model.

HOWEVER ... it is generally considered good practice to have your  
Catalyst::Model a thin wrapper around a model that will work  
standalone.  This means for example that you can use your model  
standalone with a shell script for example.  In the case of Connotea  
this is probably not nescessary, but this thin wrapper around a fat  
model approach is still well worthwhile because it will make the code  
much more modular.  Bibliotech does pretty heavy lifting in the  
database department, and a standalone database only model would make  
better abstraction, and greatly reduce cognitive overhead for your  
developers.

Having said this I had a look at the docs for DBIx::Class::Cursor and  
DBIx::Class::Cursor::Cached and it looks very much to me like you are  
getting transparrent caching in the deal and wouldn't need  
ACCEPT_CONTEXT or equivalent.  The next bit is crucial:  your caching  
stuff has no test coverage.  moving to CDBICompat, and the  
cursor::cached engine buys you all thier test coverage for free, with  
better separation and better debugging tools.  I'd do this myself,  
but it's not my code, so I'd rather that someone who was familliar  
with it did the reverse plumbing.  Once that's done I'm likely to get  
around to doing a demo refactor of one of the methods to DBIC.

Oh.  I made a #connotea channel on irc.perl.org - although our  
timezones are off, I'm just going to idle there until someone else  
turns up :-)

[Connotea-code-devel] more on DBIx::Class::CDBICompat

[Kieren D. <ki...@di...>]

My impatience got the better of me, so I had a shot with porting the  
cdbi code to DBIx::Class::CDBICompat.  I'm reliably informed that  
once DBIx::Class 0.0810 is released in a few weeks, moving from CDBI  
to DBIC::CDBICompat will be a simple matter of swapping out the use  
base 'Class::DBI' to use base 'DBIx::Class::CDBICompat';

Then you get the DBIC feature set on top of your cdbi classes.  I'm  
reliably informed (via irc) that Bibliotech::DBI is too highly  
customised for a straight port to dbic. However you still get the  
DBIC_TRACE=1 sql diagnostics (including for inserts), and the more  
transparent and swappable caching possibilities for very little effort.

So I tried:

$ mkdir connotea-dbic
$ cd connotea-dbic
$ mkdir lib
$ mkdir t
$ cp -R /path/to/connotea-code/bibliotech/Bibliotech/DBI* lib/

chuck a nice little test file in t/01-load-all.t (thanks to matt  
trout :-) ):

use strict;
use warnings;
use Test::More ();
Test::More::plan('no_plan');
use Module::Pluggable::Object;
my $finder = Module::Pluggable::Object->new(
                search_path => [ 'Bibliotech' ],
              );
foreach my $class (sort $finder->plugins) {
   Test::More::use_ok($class);
}

edit up the bibliotech config so the dsn is correct and go:

prove -l t

boom.  the model is too tightly coupled to the controller and view.


so this is an example of a problematic line:

my $time = $Bibliotech::Apache::QUICK{DB} ||  
$Bibliotech::Apache::MEMCACHE->get('DB');

I don't quite understand why I'm seeing dependencies to  
Bibliotech::Component in the test ouptut too, but that's another  
(probably more minor) problem.

I think all caching needs to be removed first then reintroduced  
transparently.  Given that refactoring to DBIC::CDBICompat is almost  
certain to be easier and more productive in the long
run that doing a pure cdbi refactor, and that DBIC::Cursor::Cache  
will work with the CDBICompat code (in the remote chance it doesn't  
the community will certainly be responsive to failing test case and/ 
or patches).  There's the path of least resistance to fixing this  
problem.  Also a good time to start writing tests for  the model - I  
can see why there aren't tests at the moment with the tight coupling.

Once that's done, it becomes straightforward to port some of the  
larger methods to DBIC without the compat layer, and you start to  
gain the huge benifits of easily interrogated classes and chainable  
resultsets.

Going down this track is going to increase the chances of finding  
committed volunteers (volunteers of some sort or another are the  
lifeblood of open source software).  The less cognitive overhead your  
developers have while dealing with the software, the more likely  
they'll stick with it, and the more likely you are to get quality code.

Re: [Connotea-code-devel] MVC.

[Kieren D. <ki...@di...>]

On 11 Aug 2007, at 02:10, Martin Flack wrote:

> Kieren, thanks for these comments.
>
> The bulk of our work is on citation/import/export, speed  
> optimization, user features, etc. so I am not averse at all to  
> having some of the structural points reexamined, and you're  
> probably correct that we haven't thought about them questioningly  
> in a long time. If this is an issue for newcomers I'd like to be  
> aware of it.

I had a good long look at connotea about a year ago, but with my  
limited time, it was just too hard for me.  I would really like to  
use it again for a new project now, but I can't make full use of it  
for now, for the reasons I gave before.

I still really want to persuade you to refactor to DBIx::Class, and I  
think that this process may even be quicker than refactoring the cdbi  
code on its own, and certainly more developer/community  friendly.   
Bear in mind that many DBIx::Class developers are former cdbi  
developers and its design is informed by a knowledge of cdbi's  
limitations.  In connotea the model is the largest software  problem  
to be solved, and while you've solved it well in many ways, it's very  
hard for casual developers to pick it up and run with it.  I think  
DBIx::Class would provide much of a solution to this - the biggest  
feature in my book being DBIC_TRACE=1 myscript.pl producing all the  
sql used to STDERR

I've attached a tarball of some DBIC/Catalyst::Model code from the  
LinkMine (social bookmarking) project which I referred to in an  
earlier email.  It's not working code as such, and I'm not completely  
convinced that it follows best practice in all cases, but it shows  
the general idea.  I've also greped for all instances where the model  
is called in the controller code too.  I had to do some slightly  
exotic stuff to get a mysql database schema.  Basically I took the  
DBIx::Class schema declared in LinkMSchema (which was written for  
postgres originally) and ran the following script:

#!/usr/bin/perl
use warnings; use strict;
use LinkMSchema; # DBIx::Class schema
system('sqlite3 db'); # eugh, hack needs connect_info a blank sqlite  
db will do
my $schema = LinkMSchema->connect( "dbi:SQLite:db");
$schema->create_ddl_dir('MySQL', undef, './');

which provides me with the sql for a mysql schema.  I did lose a  
couple of triggers, but that's not important for illustrative purposes.

I found the horrible query you mentioned in Bibliotech::DB.pm .  You  
don't have to work around the limitations of DBIx::Class to implement  
this, it knows it's own limitations.  Here's the perldoc that sums up  
the situation with Connotea perfectly:  http://search.cpan.org/ 
~mstrout/DBIx-Class-0.08005/lib/DBIx/Class/Manual/ 
Cookbook.pod#Arbitrary_SQL_through_a_custom_ResultSource

Back in the day when I was using CDBI I didn't use it much because I  
found it so hard going, with uninformative errors, and documentation  
that I found difficult to follow.  It also has a habit of deep  
resucsion - you can demonstrate this with the connotea code base by  
adding Sub::WrapPackages into an appropriate place for debugging (I  
put it in Bibliotech::Apache).  Then DBIx::Class came along which has  
learned from the mistakes of CDBI.  Bear in mind my database usage is  
generally pretty limited, with little or no need for optimisation on  
my side so a lot of the DBIC code I write looks like this (in memory  
loading of the schema):

#!/usr/bin/perl -w
use DBIx::Class::Schema::Loader qw/make_schema_at/;
make_schema_at( 'MyDB', {relationships =>1, debug =>0, auto_update =>1},
                 [ 'dbi:mysql:database=mydb', 'user', 'pass'],);
my $schema= MyDB->connect();
my $rs = $schema->resultset('Table')->search({});
while (my $rec = $rs->next()) {
     # do stuff
}


>> 2.  Taking your dispatcher code, getting it working independent  
>> of  apache (basically factoring out $r into utility functions).   
>> Next up,  delegating the dispatching logic and content generation  
>> logic below  and bending it into catalyst logic.  I really think  
>> this isn't going  hard for someone familiar with the codebase,  
>> although it would be  tricky for me.  What this buys you is a  
>> clearly understood dispatch  process that's very easily understood  
>> by a large and active developer  community.
>
> $r is basically only handled in Apache.pm. You can write a test  
> script that creates a Bibliotech::Command or uses  
> Bibliotech::Parser to create one. Bibliotech::Fake was created for  
> this purpose.
>
> Having said that, you are correct it would be good to add more  
> abstraction to make this easier, and make command-line uses easier.
>

Indeed.  Also any example command line scripts would be much  
appreciated this would also help to begin to address the mod_perl  
dependency, which is a problem for widespread use of your code.  The  
mod_perl dependency is a show stopper for me - if I get a working  
standalone model with example code, I'm much more likely to hack at  
it (although CDBI still leaves me inclined to avoid unless there's  
lots of clear example code).


> things like using a cache engine ie memcached should be  
> transparent  and trivial to remove for users who don't need that  
> feature with DBIC  (c.f.  tight couplling and  real maintenance  
> headache for the  equivalent CDBI code).
>
> It is abstracted in Bibliotech::Cache so maybe just a  
> Bibliotech::Cache::None or Bibliotech::Cache::Memory gets added to  
> the project? A little bit of the caching is useful in the same  
> request.

If you're going to do that, make life easy on your developers and aim  
for total transparrency - so a null cache, then a slightly lighter  
cache if that proves necessary. Then memcached.

>
> Do you happen to know of any public Catalyst projects that are  
> required to use server-wide caching to handle the traffic load? So  
> we could see their programming model for the cache interface? That  
> would be interesting.
>

There are a few catalyst plugins on CPAN  that provide simple  
caching.  These can be modified for more complex use-cases of  
course.  These include component caching (see Catalyst::Plugin::Cache  
which has a bare bones catalyst app as part of its test suite), page  
cache (Catalyst::Plugin::PageCache - again with good app based test  
coverage).  Catalyst::Plugin::Cache::Memcached lacks this  
unfortunately. The core of caching with DBIx::Class (with the  
swappable backends) is DBIx::Class::Cursor::Cached.  TT also does  
some memory caching of its own.

Something else that catalyst buys you is the ready-rolled ability to  
distribute your app as a CPAN dist.  This makes installation pretty  
easy assuming you have your Makefile.PL set out properly:

$ perl Makefile.PL
$ make installdeps # or sudo make installdeps

Also as far as catalyst goes, there's a well developed set of (again  
multiple backend)  authentication/authorisation code.  However it's  
obvious that the data store for your auth/authz is in mysql.

(mail cc'd to your personal address in case the tarball gets stripped  
by sourceforge).

Kieren

Re: [Connotea-code-devel] MVC.

[Martin F. <ma...@ne...>]

Kieren, thanks for these comments.

The bulk of our work is on citation/import/export, speed optimization, 
user features, etc. so I am not averse at all to having some of the 
structural points reexamined, and you're probably correct that we 
haven't thought about them questioningly in a long time. If this is an 
issue for newcomers I'd like to be aware of it.

Kieren Diment wrote:
> 1.  Getting the database code working independently of the web  
> application and calling it a well separated model.

Your WIAB project looks file-based; I'd be interested in a similarly 
small SQL project.

> 2.  Taking your dispatcher code, getting it working independent of  
> apache (basically factoring out $r into utility functions).  Next up,  
> delegating the dispatching logic and content generation logic below  
> and bending it into catalyst logic.  I really think this isn't going  
> hard for someone familiar with the codebase, although it would be  
> tricky for me.  What this buys you is a clearly understood dispatch  
> process that's very easily understood by a large and active developer  
> community.

$r is basically only handled in Apache.pm. You can write a test script 
that creates a Bibliotech::Command or uses Bibliotech::Parser to create 
one. Bibliotech::Fake was created for this purpose.

Having said that, you are correct it would be good to add more 
abstraction to make this easier, and make command-line uses easier.

I suppose at the moment I am perhaps so familiar with the codebase I can 
write a test script "too quickly."

>> I think you'd get your CMV separation as good as any application  
>> can get with these steps:
>>
>> - removing the *_content() methods and supporting items and making  
>> them subclasses of the Bibliotech::Page and Bibliotech::Component  
>> objects with a content() method. That just seemed complicated
>>
>> - removing the *_content() methods and supporting items from  
>> Bibliotech::DBI objects and putting them into similarly separated  
>> subclasses or have another class that accepts objects and renders  
>> them or something like that.
>>
>> It just seemed like that increased syntactical complexity and class  
>> complexity in order to create separation that we didn't think we  
>> needed when the codebase was smaller.
>>
> 
> mmm.  I really think that's called technical debt

Perhaps an explicit CMV remodel like this will turn out to be the first 
suggestion offered by you that we would try to implement then.

>> I'm actually interested in seeing a fairly big application written  
>> in Catalyst to see how you guys do it.
> 
> right.  The really big ones tend to be closed source.  A prominent  
> one ( Vox from 6apart):  http://tokyo2007.yapcasia.org/sessions/ 
> 2007/02/everything_vox.html

I certainly will take a look when I have time.

> and a little experiment:
> 
> /tmp$ du -h -d 1 connotea-code-1.7.1
> 1.0M    connotea-code-1.7.1/bibliotech
> 1.0M    connotea-code-1.7.1

I don't understand what your actual point here is though.

> Here's my Website in a Box which I wrote while learning to provide a  
> complete and useful calalyst app with good test coverage and good  
> programmer documentation.  The dispatch logic here is very simple.   
> http://code.google.com/p/websiteinabox (codebase = 1.3M).  It has  
> some flaws (the controller is too thick in places and the model too  
> thin mainly).

Interesting.

> Now onto the Model:
> 
>> Unfortunately we are not even within the bounds of Class::DBI  
>> because Bibliotech::Query does a lot of query building itself,  
>> although I am interested in revamping it because it is the most  
>> difficult part of the codebase. So we'd probably have to do a  
>> general refactoring before we even switched.
>>
> 
> Right.   DBIC was made because projects outgrew CDBI very quickly in  
> exactly this manner. In this case I'd really recommend doing a  
> concurrent refactor to DBIx::Class to get a model that works  
> independently of the app.  Getting a starting schema with good  
> relationships coverage really is as simple as installing DBIx::Class  
> and doing something like:

I'm not convinced your examples are as aggressive as we are in the SQL 
area. Just for giggles, I'm going to post a query that collects the data 
for /user/martin while logged in as martin. This will give you an idea 
of the consolidation that was necessary to collect the data for one page 
of Connotea in a reasonable number of seconds. A similar set of queries 
to collect all the requisite data was over 800 queries (really) when 
leaving Class::DBI to get the data itself so this is what it looks like 
now after packing it into one monster query that runs quickly. I'll post 
in a separate email.

I'm not saying you're wrong, just take a look if you like.

> You've got a lot of sql you can then port back in by using the scalar  
> ref feature in SQL::Abstract, and from there you can wind back to  
> something a bit easier to maintain using core dbic functionality  
> wherever possible.

BTW, we did use SQL::Abstract, so we do know about it.

> chainable resultsets are a feature everybody finds extremely useful:
> 
> my $schema  = DB->connect();
>   my $resultset = $schema->resultset('Foo')->search(\% 
> scary_search_condition, \%search_attributes);
> my $narrowed_resultset = $resultset->search(\%some_more_search_logic);
> while (my $foo = $narrowed_resultset->next) { # you only hit the  
> database at this point in the chaining of the resultset }

Ok. I'll have to study that class.

> things like using a cache engine ie memcached should be transparent  
> and trivial to remove for users who don't need that feature with DBIC  
> (c.f.  tight couplling and  real maintenance headache for the  
> equivalent CDBI code).

It is abstracted in Bibliotech::Cache so maybe just a 
Bibliotech::Cache::None or Bibliotech::Cache::Memory gets added to the 
project? A little bit of the caching is useful in the same request.

Do you happen to know of any public Catalyst projects that are required 
to use server-wide caching to handle the traffic load? So we could see 
their programming model for the cache interface? That would be interesting.

I'd note we cache in M and V stages in Connotea. Query results as 
constituted objects as well as component HTML and whole pages (HTML or 
otherwise) are cached, for visitors, and for individuals (since privacy 
affects everything we show in Connotea).

> DBIx:Class ought to see the following schema from the connotea db  by  
> default (ie relationships pre-specified for you in the generated  
> code) - http://www.uow.edu.au/~kdiment/schema.png (generated by  
> sqlfairy which DBIC uses for some purposes). I can't guarantee this,  
> but if not it will be close.

Sure.

Cheers,
Martin

[Connotea-code-devel] example sql

[Martin F. <ma...@ne...>]

Hope this forward makes it to the mailing list.

This is something I thought I'd share as part of a discussion with 
Kieren. If anyone has better approaches for what we're accomplishing (in 
the speed we're accomplishing it) I'm humble enough to listen. ;-)

I'll note one thing I am aware of is that FOUND_ROWS() is available to 
us in the newer versions of MySQL. I tested the real speed and it wasn't 
much different in practice so I haven't given it any priority at all yet.

Something that might not be obvious to people not working on the project 
is how intensive the privacy feature is on the SQL. This is for a logged 
in user - a visitor gets a slightly simpler query with respect to 
privacy because we have a cheat column for them, but it still has to 
check private_until because that changes with time. A cron job could 
update a more sophisticated cheat column but we haven't done that yet.

[Connotea-code-devel] Another substantial catalyst example

[Kieren D. <ki...@di...>]

OK, I forgot possibly the most significant open source catalyst  
application (recepient of perl foundation grant):  http:// 
handelframework.com/


Note that this project uses Perl::Critic to maintain coding standards.

Kieren

[Connotea-code-devel] MVC.

[Kieren D. <ki...@di...>]

On 10 Aug 2007, at 00:33, Martin Flack wrote:

> A brief sketch of the current dispatch:
>
> Bibliotech::Apache runs the dispatch. Bibliotech::Parser  
> understands the URI passed in, and returns a Bibliotech::Command  
> object. An appropriate Bibliotech::Page object is created, which  
> provides a last_updated() method and an html_content() method that  
> return results (similarly, it is ris_content() or rss_content()  
> etc. for other formats). The html_content() method generally uses  
> TT and Bibliotech::Query and various Bibliotech::Component's and  
> some help from Bibliotech::DBI layer and Bibliotech::Util.  
> Bibliotech::DBI is the base class for all objects in the database,  
> and yes they all have an html_content() as well. Actions such as  
> adding and editing have root functions in Bibliotech.pm since they  
> coordinate across many modules.
>

Riight

I think that you could do a fairly straightforward refactor to  
catalyst by:

1.  Getting the database code working independently of the web  
application and calling it a well separated model.

2.  Taking your dispatcher code, getting it working independent of  
apache (basically factoring out $r into utility functions).  Next up,  
delegating the dispatching logic and content generation logic below  
and bending it into catalyst logic.  I really think this isn't going  
hard for someone familiar with the codebase, although it would be  
tricky for me.  What this buys you is a clearly understood dispatch  
process that's very easily understood by a large and active developer  
community.

> I think you'd get your CMV separation as good as any application  
> can get with these steps:
>
> - removing the *_content() methods and supporting items and making  
> them subclasses of the Bibliotech::Page and Bibliotech::Component  
> objects with a content() method. That just seemed complicated
>
> - removing the *_content() methods and supporting items from  
> Bibliotech::DBI objects and putting them into similarly separated  
> subclasses or have another class that accepts objects and renders  
> them or something like that.
>
> It just seemed like that increased syntactical complexity and class  
> complexity in order to create separation that we didn't think we  
> needed when the codebase was smaller.
>

mmm.  I really think that's called technical debt

> I'm actually interested in seeing a fairly big application written  
> in Catalyst to see how you guys do it.

right.  The really big ones tend to be closed source.  A prominent  
one ( Vox from 6apart):  http://tokyo2007.yapcasia.org/sessions/ 
2007/02/everything_vox.html

and a little experiment:

/tmp$ du -h -d 1 connotea-code-1.7.1
1.0M    connotea-code-1.7.1/bibliotech
1.0M    connotea-code-1.7.1

/tmp$ cd ~/Deskop/dev/Catalyst/trunk/examples # (http:// 
dev.catalyst.perl.org/repos/Catalyst/trunk/examples/)

~/Desktop/dev/Catalyst/trunk/examples$ du -h -d1
# I've edited out the ones I think are of no use to you of I don't  
know anything about
136K    ./AdventREST # example REST app
616K    ./CatalystAdvent # lots of example driven documentation
152K    ./CatTube # youtube integration
216K    ./ChainedEg # innovative dispatch method example - may be  
relevant to connotea may not
132K    ./Cheer # very simple tutorial app
144K    ./GeoCat # geotagging
488K    ./InstantMessenger # neat IM program - old so contains some  
deprecated approaches
156K    ./JQChat # minimal DBIC/Query chat program
152K    ./MiniMojoDBIC # minimal DBIC/Prototype wiki
172K    ./OpenID # openID integration
412K    ./ServerDB # server management software
3.2M    ./SmokeServer # server for testing software?
40K    ./Streaming # simple streaming app
352K    ./Tutorial # reference implementaiton for the catalyst  
tutorial in Catalyst::Manual::Tutorial
9.2M    .

Here's my Website in a Box which I wrote while learning to provide a  
complete and useful calalyst app with good test coverage and good  
programmer documentation.  The dispatch logic here is very simple.   
http://code.google.com/p/websiteinabox (codebase = 1.3M).  It has  
some flaws (the controller is too thick in places and the model too  
thin mainly).

Now onto the Model:

> Unfortunately we are not even within the bounds of Class::DBI  
> because Bibliotech::Query does a lot of query building itself,  
> although I am interested in revamping it because it is the most  
> difficult part of the codebase. So we'd probably have to do a  
> general refactoring before we even switched.
>

Right.   DBIC was made because projects outgrew CDBI very quickly in  
exactly this manner. In this case I'd really recommend doing a  
concurrent refactor to DBIx::Class to get a model that works  
independently of the app.  Getting a starting schema with good  
relationships coverage really is as simple as installing DBIx::Class  
and doing something like:

perl -MDBIx::Class::Schema::Loader=make_schema_at,dump_to_dir:/path/ 
to/lib -e  \
'make_schema_at("MySchemaName", {relationships => 1},  
["dbi:mysql:dbname=db", "user", "pass"])'

You've got a lot of sql you can then port back in by using the scalar  
ref feature in SQL::Abstract, and from there you can wind back to  
something a bit easier to maintain using core dbic functionality  
wherever possible.

chainable resultsets are a feature everybody finds extremely useful:

my $schema  = DB->connect();
  my $resultset = $schema->resultset('Foo')->search(\% 
scary_search_condition, \%search_attributes);
my $narrowed_resultset = $resultset->search(\%some_more_search_logic);
while (my $foo = $narrowed_resultset->next) { # you only hit the  
database at this point in the chaining of the resultset }

things like using a cache engine ie memcached should be transparent  
and trivial to remove for users who don't need that feature with DBIC  
(c.f.  tight couplling and  real maintenance headache for the  
equivalent CDBI code).

DBIx:Class ought to see the following schema from the connotea db  by  
default (ie relationships pre-specified for you in the generated  
code) - http://www.uow.edu.au/~kdiment/schema.png (generated by  
sqlfairy which DBIC uses for some purposes). I can't guarantee this,  
but if not it will be close.

As you can see I can continue this conversation for ages :-)

Kieren

Re: [Connotea-code-devel] feedback

[Martin F. <ma...@ne...>]

Hi Kieren,

A couple quick responses.

Kieren Diment wrote:
> 1.  Controller
> 
>   Porting to Catalyst provides the framework for this, with sane url/ 
> action dispatching (I couldn't work out how this works looking at the  
> connotea code base  with my limited available time.  This is another  
> problem).  Using the catalyst dispatcher makes it very quick for new  
> developers to get their head around the flow of logic within the  
> application (the controller part).  The main task here is refactoring  
> the mod_perl handler dispatching logic into something catalyst  
> compatible.  I've seen people doing this a few times, and it should  
> be pretty quick and straightforward for someone familliar with the  
> existing mod_perl code base.  An experienced mod_perl hacker (which  
> I'm not) from the catalyst project was going to have a look at the  
> connotea code base for me to tell me if my assumptions above are  
> correct, but because there's no code viewable in a web browser, this  
> proved too much friction ... for now.  (Here illustrates a major  
> benifit of open source software communities - a lot of my comments  
> don't come from my expertise, but from knowing how to ask the right  
> questions on the appropriate irc channel).

A brief sketch of the current dispatch:

Bibliotech::Apache runs the dispatch. Bibliotech::Parser understands the 
URI passed in, and returns a Bibliotech::Command object. An appropriate 
Bibliotech::Page object is created, which provides a last_updated() 
method and an html_content() method that return results (similarly, it 
is ris_content() or rss_content() etc. for other formats). The 
html_content() method generally uses TT and Bibliotech::Query and 
various Bibliotech::Component's and some help from Bibliotech::DBI layer 
and Bibliotech::Util. Bibliotech::DBI is the base class for all objects 
in the database, and yes they all have an html_content() as well. 
Actions such as adding and editing have root functions in Bibliotech.pm 
since they coordinate across many modules.

I think you'd get your CMV separation as good as any application can get 
with these steps:

- removing the *_content() methods and supporting items and making them 
subclasses of the Bibliotech::Page and Bibliotech::Component objects 
with a content() method. That just seemed complicated

- removing the *_content() methods and supporting items from 
Bibliotech::DBI objects and putting them into similarly separated 
subclasses or have another class that accepts objects and renders them 
or something like that.

It just seemed like that increased syntactical complexity and class 
complexity in order to create separation that we didn't think we needed 
when the codebase was smaller.

I'm actually interested in seeing a fairly big application written in 
Catalyst to see how you guys do it.

> 2.  Model
> 
> DBIC is much more flexible and much faster than CDBI.  A port would  
> again reduce technical overhead.  However if you can get the CDBI  
> model working independently of the mod_perl app, it becomes usable  
> within catalyst, and you can defer the decision to port while  
> improving the structure of the codebase.  However, the tight coupling  
> of the cdbi models with memcached is another impediment to casual  
> hacking on connotea.  I suspect that porting to DBIC would make life  
> a lot easier for junior and senior developers alike, and be the  
> easiest way to get a web-app-independent model available as well.   
> Thanks for reminding me about the sql trace stuff you added.  It was  
> a while ago, but not being able to see the do() stuff was  a show  
> stopper for me I'm afraid.

Unfortunately we are not even within the bounds of Class::DBI because 
Bibliotech::Query does a lot of query building itself, although I am 
interested in revamping it because it is the most difficult part of the 
codebase. So we'd probably have to do a general refactoring before we 
even switched.

> 3.  View
> 
> Based on your feedback, if using  catalyst, you'd probably want to  
> make a souped up Catalyst::View::ConnoteaTT based on  
> Catalyst::View::TT.  Maybe also  make a  Catalyst::View::RDF as  
> well.  Improving the architecture to something more MVC is going to  
> make all these little components much more modular and easier for new  
> (junior and senior) developers alike to understand.
> 
> A core problem here is that with the current architecture/libraries  
> of connotea you don't have very good ability to leverage the perl  
> community to help with development.  You should also upload the  
> connotea perl api libraries to CPAN so that connotea becomes more  
> visible to the perl community at large, and make a post to  
> use.perl.org once the new version is out with it's vc repository.
> 
> I didn't suggest Rails because Catalyst is far superior both in  
> library support and that it's much less opinionated, and thus has the  
> flexibility that connotea requires. :-D

All noted. Thanks.

Martin

[Connotea-code-devel] downscaling

[Kieren D. <ki...@di...>]

On 9 Aug 2007, at 21:16, Mulvany, Ian wrote:

> before telling you that it isn't going to happen I am going to have  
> a long
> think about this. I can say that something like this is not going  
> to happen
> soonish, but as you intimated in a later mail if you wanted to  
> prove the
> possibility in principle we would have a serious look at that.


Thanks for this.  If there is one critical thing, it's the lack of  
downscalability.  Needing a "dedicated big rig" with pretty hefty  
system requirements (i.e. mod_perl and and memcached) is the largest  
factor that stops me from working on this code base at the moment.

Looking forward to the new code.  Would it be possible for distribute  
a small sample mysql database for hacking/testing purposes with the  
new code please?

Re: [Connotea-code-devel] Connotea-code-devel Digest, Vol 16, Issue 8

[Mulvany, I. <i.m...@na...>]

Hi Kieren,

I'm responsible for all aspects of Connotea, so I will step up next.
Thanks for the feedback. I've read through all of the ongoing discussion, so
I'll just give a short overview of where we are at the moment and where we
want to be going, then I will reply to some specific points.

We are still somewhat in a state of transition as I took over this project
earlier this year. We are also transitioning from CVS to darcs in a hope
that this will make maintaining development easier for us.

I would say that it is a fair comment to point out that our focus has
strongly been on the user rather than the developer, however I recognize
that the long term health of the project is dependant upon developers being
able to work on this with us for aspects of the service that they would like
to see extended or improved. We are in discussion with a number of people
who are interested in principle in taking on some work with an eye to
extending the feature set of connotea. This is important for us and that we
have to right type of system in place to support this is important for me.

We have not had a new release of the codebase for about a year, and my first
priority to get the updated version out, and then to augment this with
example code and documentation. I hope that point we can take stock and see
what other more critical changes might need to take place.

So that's kind of our current state.


>> 
>> Thanks for your email. I'm responsible for most of the codebase so
>> I will step into the firing line first. ;-)
>> 
>> Kieren Diment wrote:
>>> I've been wanting to write this email for a while, and you can
>>> consider me a strong supporter of what Connotea/NPG  is trying to
>>> do.  However there are some major problems that I think need to
>>> be  addressed if connotea is to become a self sustaining open
>>> source  project with a sustainable developer community.
>> 
>> Again thanks. I'm aware of most of what you raise, and in fact have
>> discussed them with folks, but it doesn't hurt to acknowledge on
>> the devel list.
>> 
>> Actually this works out nicely since we are about to release a new
>> version of Connotea Code literally any day now, and we are
>> committed to posting a public darcs repository and inviting more
>> public support. I'm actually thinking you have seen the new code by
>> some of your comments.
>> 
>>> A bit about me:  I'm a social researcher and perl hacker.  Most
>>> of  the perl I write is for research data management, but I do
>>> bibliographic stuff, web robots and occasional web applications
>>> programming too.  I've been a significant contributor to the
>>> Catalyst  web application framework (mainly with the documentation
>>> and example  code).  I'm computer literate, and I've used mod_perl
>>> enough to have  a good idea of what irritates me about it.  I
>>> think that Catalyst is  the natural successor for 95% of the
>>> mod_perl or Apache::Registry  scripts out there.
>> 
>> I've looked at Catalyst but I've not used it.
>> 
>>> 1.  No test suite.  Lack of unit/behaviour tests along with the
>>> impoverished debugging environment with mod_perl make for painful
>>> development. [ SOLUTION:  Port to catalyst with built in testing
>>> utilities and trivial support for perl -d ]
>> 
>> The new version of Connotea Code sports a fledgling test suite. It
>> doesn't shoot for code coverage, just to test the basic functions.
>> It's an area I'd like to strengthen.
>> 
>> I concur on the mod_perl debugging situation.
>> 
>>> 2.  Class::DBI + memcached.  These seem to be very tightly
>>> coupled.   Class::DBI is flawed software, and while it has proved
>>> very useful,  it clearly has severe limitations.  One example is
>>> the difficulty of  inspecting the sql that's it's generating.
>>> It's main problem is a  general lack of transparency and
>>> implicitly generated code.  CDBI   "died" as an ongoing open
>>> source project about a year ago.  The  original author of CDBI's
>>> last project resulted in him fixing  DBIx::Class' CDBI
>>> Compatibility layer  so he could port his client's  code from
>>> CDBI  to DBIx::Class.
>> 
>> Accurate observation. At the time it seemed like the best way to
>> provide abstractions that made the model possible.
>> 
>> I'd add a point that you didn't make, which is that
>> Bibliotech::Query is  very complicated, not very clear, and the
>> queries it generates are huge, but to some extent it's by necessity.
>> 
>> For the type of thing like Connotea, Class::DBI is way too slow
>> left to its own devices - a page was taking 800 queries to pull up
>> all the pieces of data, and we solved that with a scary-looking but
>> fast-executing query. Another problem might be that it was designed
>> to be a very flexible query engine, and as it turns out that
>> flexibility is obscuring some of the functionality, i.e. tightening
>> the restraints might make prettier source code.
>> 
>> What we got from Class::DBI was a convenient abstraction that
>> database rows would be objects that could relate to each other.
>> Beyond that we've extended the heck out of it, and probably not
>> canonically, although in our defense Class::DBI is only designed to
>> have a canonical usage for the theoretical database model, not a
>> whole application.
>> 
>> The new version of Connotea Code has a Bibliotech::Query with even
>> more speed optimizations but again it's another layer of complexity.
>> 
>>> [SOLUTION:  implied by the last comment DBIx::Class (DBIC)  is
>>> the  natural successor to CDBI, originally developed by the author
>>> of  CDBI::Sweet.  DBIC + memcached has production users.
>>> Discussion on  IRC  (#dbix-class on irc.perl.org) indicates to me
>>> that swappable  caching engines (including a null cache for
>>> debugging) ) should be  trivial and transparent with a DBIC based
>>> data model.  The killer  dbic feature for me is that running your
>>> dbic script like so:   "DBIC_TRACE=1 perl run_my_script.pl" gives
>>> you the exact sql being  passed to the dbic classes.  This outputs
>>> to standard error, thus  making debugging hugely easier.  As
>>> noted, there's a CDBICompat layer  to ease transition from CDBI to
>>> DBIC.  CDBICompat has a more  extensive test suite than CDBI for
>>> good measure.  Finally DBIC tends  to result in much more efficent
>>> SQL than CDBI and swapping out  database engines (eg mysql to pg
>>> to sqlite) is much easier - I've  seen this done trivially from pg
>>> to sqlite for a database  representing a directed acyclic graph. ]
>> 
>> I've got SQL tracing - everything except do() calls and without
>> question-mark translation - if you call Bibliotech::DBI-
>>> activate_warn_sql(). In mod_perl this sends up in Apache's error_log.
>> 
>> Again I've hard of DBIx::Class but was not aware that it offered
>> any major benefits.
>> 
>>> 3.  Template toolkit.  The templates in connotea seem to be
>>> populated  with weird coderefs that make debugging/interrogation
>>> even more  difficult.  TT is one of the best templating solutions
>>> out there, but  connetea seems to misuse it. [SOLUTION:  Port to
>>> explicit templates  resident in files, using PROCESS, INCLUDE and
>>> MACRO blocks where  appropriate.  This should be fairly simple to
>>> do concurrently with  the rest of the catalyst/dbic port. ]
>> 
>> Anything that looks like misuse of TT is probably a result of
>> yielding functionality from our original Component system to TT,
>> where the rest of the Component system is still there, and which
>> was more capable than what we perceived TT could do, and we needed
>> to preserve functionality.
>> 
>> It may be that TT could do more than we realized, or could not then
>> but can now, and in that case it would be, as you point out, better
>> to use the native TT functionality rather than homegrown
>> functionality. I agree on principle.
>> 
>> As an example, our Component system was designed to put all the
>> logic for a component in one place, and avoid expensive
>> recalculation. To that end, a component's result is not just a
>> snippet of HTML like an INCLUDE, but an object that contains HTML
>> parts that can be put in different places on the calling web page,
>> as well as the Javascript for the <head> and Javascript for <body
>> onload=""> that facilitate the component, so if you want to, say,
>> position the cursor in the first form field, you can have the
>> component return the HTML form and that Javascript together.
>> 
>> The new Connotea Code release has a section in the README to cover
>> the TT function calls which should explain things better.
>> 
>>> 4.  Connotea doesn't scale down well.  I've used it on an ibook
>>> g4  for testing (performance verging on reasonable), and a pentium
>>> 3  linux machine with 128mb of RAM where performance was
>>> unacceptable.  [SOLUTON:  port to catalyst where fast_cgi,
>>> mod_perl and other more  exotic engines are all viable solutions.
>>> Removing the mod_perl  depencency opens up shared hosting
>>> possibilities]
>> 
>> We used to have a bibliotech.cgi that could do exactly that (using
>> Apache::Emulator) but we dropped it because NPG had no need for it
>> and it was after all just a hack. I'd be happy to share it with you
>> if you are interested, but the added memcached requirement is the
>> main obstacle I think.
>> 
>>> There are some excellent things about connotea (auto-import and
>>> the  database schema being two big examples), but the above are
>>> show  stoppers which are going to cause huge problems for the
>>> sustainability of the project.  The usability for end users is
>>> great,  but as far as programmer-usability goes connotea needs
>>> major  improvements otherwise it doesn't have a future as a viable
>>> open  source project.  Personally I'd like to hack in storage of
>>> pdf/other  fulltext into the database, but I can't do this in it's
>>> current  state.  I can also see why NPG won't do this themselves.
>> 
>> Thanks for the compliments!
>> 
>> At least you're not complaining it's not Ruby on Rails. ;-)
>> 
>> That's a joke, but I acknowledge that CPAN marches forwards even
>> after we have selected our libraries and yes platforms make a huge
>> difference. We selected solutions to match the engineering and then
>> we tried to move to some newer approaches as dictated by practical
>> concerns. Even moving to TT was only a direct response to needing
>> non-programmers to do editing and design work.
>> 
>>> My vision is for distributed collaborative bibliographies, which
>>> is  why I'd like so much for the project to scale down to a level
>>> where  it could be used by 2-10 researchers on shared hosting.  I
>>> think  optional  re-import back to a master connotea  would be
>>> fairly easy  to implement after the scaling down problem was
>>> addressed.
>> 
>> Your work sounds interesting.
>> 

I second this point.

>>> Part of the problem is that connotea came about at a time of
>>> great  flux during the web app programming space, and the
>>> technology to  program these things has improved massively during
>>> this time.
>> 
>> You are saying something that sounds to me like:
>> 
>> - refactoring for better library support would make easier development
>> 
>> I'd also add:
>> 
>> - like all things, more man-hours on the codebase for general
>> refactoring or more test suite scripts would make easier
>> development, keeping the same libraries even
>> 
>> - refactoring to remove some early assumptions would make a cleaner
>> codebase as well
>> 
>> ...and I'd argue that those two points are just as important.
>> 
>>> Unfortunately it leaves connotea with more technical debt.
>> 
>> Well, I think you may be being a bit hard on us here. ;-)
>> 
>> This is a classic challenge facing companies with working code;
>> whether or not to spend time refactoring things.
>> 
>> When allocating sparse resources carefully, while being customer
>> driven, a lot of the push is inevitably on keeping it running and
>> adding features rather than revising the abstractions for something
>> that already works.
>> 
>> Further, things that may make programming easier but don't actually
>> lower the complexity level of the codebase are not as appealing in
>> general as refactoring projects; e.g. switching from Class::DBI to
>> DBIx::Class is not going to make it easier to understand, (maybe)
>> just easier to work with. The concepts don't change much, you still
>> have an abstraction layer that provides SQL support. You and I can
>> appreciate that, but companies generally are interested in
>> refactoring code when it can introduce junior programmers, not just
>> make it easier for the senior programmers.
>> 
>> Having said that, your suggestions that we may be doing something
>> in a non-standard manner, if they can be done in a standard manner,
>> I do consider bugs to fix, as we should do things the same way as
>> everyone else where possible.
>> 
>> If Class::DBI -> DBIx::Class is thought of as that type of problem,
>> then I can appreciate the desire to switch over. But I'm speaking
>> here more of things like using homegrown approaches over TT
>> directives, as discussed above.
>> 

As Martin points out, the balance in allocation of resources is a delicate
one, but it's not something that we are not thinking about.

>>> I'm happy to clarify anything I've written here on request.  If
>>> the  response is going to be "isn't going to happen, sorry about
>>> that",  I'm prepared to accept that too.  As a first step I'd
>>> recommend  popping on to #catalyst at irc.perl.org and asking
>>> about porting  mod_perl apps to Catalyst.
>> 
>> I'm not the final word, and even if I was, what you've presented
>> deserves study and thought before issuing the final word. ;-)
>> 

On this point I probably am the final word, or at least close to it, and
before telling you that it isn't going to happen I am going to have a long
think about this. I can say that something like this is not going to happen
soonish, but as you intimated in a later mail if you wanted to prove the
possibility in principle we would have a serious look at that. I would
suggest waiting for the new code though.


- Ian

********************************************************************************   
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan 
Publishers Limited or its agents by means of e-mail communication. Macmillan 
Publishers Limited Registered in England and Wales with registered number 785998 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************