connotea-code-devel — Discussion and devlopment of the core Connotea code and plug-ins.

[Connotea-code-devel] More on dbix::class and social bookmarking code

[Kieren D. <ki...@di...>]

On 9 Aug 2007, at 05:00, Martin Flack wrote:

> For the type of thing like Connotea, Class::DBI is way too slow  
> left to its own devices - a page was taking 800 queries to pull up  
> all the pieces of data, and we solved that with a scary-looking but  
> fast-executing query. Another problem might be that it was designed  
> to be a very flexible query engine, and as it turns out that  
> flexibility is obscuring some of the functionality, i.e. tightening  
> the restraints might make prettier source code.

Oops, more on this.  One of my associates in Catalyst land has  
implemented a del.icio.us  clone in catalyst | dbic | tt | postgresql  
called linkmine.  It's main flaw is that it uses the "nice-idea- 
but..." library HTML::Widget - but it's most interesting from the  
database side. I've only looked at the database superficially, but  
the schema looks superficially similar (but simpler also).  With  
DBIC_TRACE=1 perl script/linkmine_server you can pretty much see what  
it's doing straight away.  The author is ZBY from CPAN (http:// 
search.cpan.org/~zby/ , http://perlalchemy.blogspot.com/)  and I'm  
sure he'd be amenable to either showing you the code or uploading the  
source to somewhere like sourceforge or google code if asked.

If you've got relationships declared in your database schema, getting  
something with the basic functions you need written to disk is as  
simple as this:

perl -MDBIx::Class::Schema::Loader=make_schema_at,dump_to_dir:/path/ 
to/lib -e  \
'make_schema_at("MySchemaName", {relationships => 1},  
["dbi:mysql:dbname=db", "user", "pass"])'

which by default will provide a good deal more than  
Class::DBI::Loader, and in a sane extendable manner.

Aside from this there's a ton of paid and community support for  
DBIx::Class (go to #dbix-class on irc.perl.org for details), and if  
there are bugs or missing features, it has a comprehnsive test suite,  
and commit to the svn is usually given out pretty freely after you've  
provided a couple of sane patches or tests.

Re: [Connotea-code-devel] feedback

[Kieren D. <ki...@di...>]

On 9 Aug 2007, at 05:00, Martin Flack wrote:

> For the type of thing like Connotea, Class::DBI is way too slow  
> left to its own devices - a page was taking 800 queries to pull up  
> all the pieces of data, and we solved that with a scary-looking but  
> fast-executing query. Another problem might be that it was designed  
> to be a very flexible query engine, and as it turns out that  
> flexibility is obscuring some of the functionality, i.e. tightening  
> the restraints might make prettier source code.

Oops,

I should mention that DBIx::Class has much better support for this  
kind of stuff.  It uses SQL::Abstract under the hood which really is  
an amazing piece of code once you get your head around it.  Passing  
raw sql into DBIC queries involves passing a scalar ref (i.e.  
\'select foo from bar') into the right part of the Resultset/SQLA  
code, but there's lots of other customisation available to make  
custom queries more convenient than this in many cases.

Re: [Connotea-code-devel] feedback

[Kieren D. <ki...@di...>]

Thanks for the quick feedback to my feedback :).  This appears to be  
quite complicated, hence the top posting (sorry ;-) )

I'd love to hack on connotea but in it's present state I can't mainly  
because of the mod_perl dependency and the tight coupling of the CDBI  
code with the application logic. I'd like to have a look at  
connotea.cgi - if it's reasonably simple I might even be able to  
demonstrate a partial catalyst port (no guarantees).

Fundamentally the connotea  codebase lacks the MVC pattern.  Moving  
to a more MVC architecture with test coverage is going to make life  
easier for everyone, senior and junior developers alike, as well as  
casual hackers like me.   So as a second iteration I'm going to deal  
with this as an MVC problem (CMV actually)

1.  Controller

  Porting to Catalyst provides the framework for this, with sane url/ 
action dispatching (I couldn't work out how this works looking at the  
connotea code base  with my limited available time.  This is another  
problem).  Using the catalyst dispatcher makes it very quick for new  
developers to get their head around the flow of logic within the  
application (the controller part).  The main task here is refactoring  
the mod_perl handler dispatching logic into something catalyst  
compatible.  I've seen people doing this a few times, and it should  
be pretty quick and straightforward for someone familliar with the  
existing mod_perl code base.  An experienced mod_perl hacker (which  
I'm not) from the catalyst project was going to have a look at the  
connotea code base for me to tell me if my assumptions above are  
correct, but because there's no code viewable in a web browser, this  
proved too much friction ... for now.  (Here illustrates a major  
benifit of open source software communities - a lot of my comments  
don't come from my expertise, but from knowing how to ask the right  
questions on the appropriate irc channel).

A major benefit of catalyst  is that perl -d script/ 
connotea_server.pl (with appropriate $DB::single=1 statements in your  
code) provides excellent debugging support.  I suspect that porting  
to catalyst would pay itself back in reduced development and training  
time very quickly.

2.  Model

DBIC is much more flexible and much faster than CDBI.  A port would  
again reduce technical overhead.  However if you can get the CDBI  
model working independently of the mod_perl app, it becomes usable  
within catalyst, and you can defer the decision to port while  
improving the structure of the codebase.  However, the tight coupling  
of the cdbi models with memcached is another impediment to casual  
hacking on connotea.  I suspect that porting to DBIC would make life  
a lot easier for junior and senior developers alike, and be the  
easiest way to get a web-app-independent model available as well.   
Thanks for reminding me about the sql trace stuff you added.  It was  
a while ago, but not being able to see the do() stuff was  a show  
stopper for me I'm afraid.

3.  View

Based on your feedback, if using  catalyst, you'd probably want to  
make a souped up Catalyst::View::ConnoteaTT based on  
Catalyst::View::TT.  Maybe also  make a  Catalyst::View::RDF as  
well.  Improving the architecture to something more MVC is going to  
make all these little components much more modular and easier for new  
(junior and senior) developers alike to understand.

A core problem here is that with the current architecture/libraries  
of connotea you don't have very good ability to leverage the perl  
community to help with development.  You should also upload the  
connotea perl api libraries to CPAN so that connotea becomes more  
visible to the perl community at large, and make a post to  
use.perl.org once the new version is out with it's vc repository.

I didn't suggest Rails because Catalyst is far superior both in  
library support and that it's much less opinionated, and thus has the  
flexibility that connotea requires. :-D

Kieren

On 9 Aug 2007, at 05:00, Martin Flack wrote:

> Hi Kieren,
>
> Thanks for your email. I'm responsible for most of the codebase so  
> I will step into the firing line first. ;-)
>
> Kieren Diment wrote:
>> I've been wanting to write this email for a while, and you can   
>> consider me a strong supporter of what Connotea/NPG  is trying to   
>> do.  However there are some major problems that I think need to  
>> be  addressed if connotea is to become a self sustaining open  
>> source  project with a sustainable developer community.
>
> Again thanks. I'm aware of most of what you raise, and in fact have  
> discussed them with folks, but it doesn't hurt to acknowledge on  
> the devel list.
>
> Actually this works out nicely since we are about to release a new  
> version of Connotea Code literally any day now, and we are  
> committed to posting a public darcs repository and inviting more  
> public support. I'm actually thinking you have seen the new code by  
> some of your comments.
>
>> A bit about me:  I'm a social researcher and perl hacker.  Most  
>> of  the perl I write is for research data management, but I do   
>> bibliographic stuff, web robots and occasional web applications   
>> programming too.  I've been a significant contributor to the  
>> Catalyst  web application framework (mainly with the documentation  
>> and example  code).  I'm computer literate, and I've used mod_perl  
>> enough to have  a good idea of what irritates me about it.  I  
>> think that Catalyst is  the natural successor for 95% of the  
>> mod_perl or Apache::Registry  scripts out there.
>
> I've looked at Catalyst but I've not used it.
>
>> 1.  No test suite.  Lack of unit/behaviour tests along with the   
>> impoverished debugging environment with mod_perl make for painful   
>> development. [ SOLUTION:  Port to catalyst with built in testing   
>> utilities and trivial support for perl -d ]
>
> The new version of Connotea Code sports a fledgling test suite. It  
> doesn't shoot for code coverage, just to test the basic functions.  
> It's an area I'd like to strengthen.
>
> I concur on the mod_perl debugging situation.
>
>> 2.  Class::DBI + memcached.  These seem to be very tightly  
>> coupled.   Class::DBI is flawed software, and while it has proved  
>> very useful,  it clearly has severe limitations.  One example is  
>> the difficulty of  inspecting the sql that's it's generating.   
>> It's main problem is a  general lack of transparency and  
>> implicitly generated code.  CDBI   "died" as an ongoing open  
>> source project about a year ago.  The  original author of CDBI's  
>> last project resulted in him fixing  DBIx::Class' CDBI  
>> Compatibility layer  so he could port his client's  code from  
>> CDBI  to DBIx::Class.
>
> Accurate observation. At the time it seemed like the best way to  
> provide abstractions that made the model possible.
>
> I'd add a point that you didn't make, which is that  
> Bibliotech::Query is  very complicated, not very clear, and the  
> queries it generates are huge, but to some extent it's by necessity.
>
> For the type of thing like Connotea, Class::DBI is way too slow  
> left to its own devices - a page was taking 800 queries to pull up  
> all the pieces of data, and we solved that with a scary-looking but  
> fast-executing query. Another problem might be that it was designed  
> to be a very flexible query engine, and as it turns out that  
> flexibility is obscuring some of the functionality, i.e. tightening  
> the restraints might make prettier source code.
>
> What we got from Class::DBI was a convenient abstraction that  
> database rows would be objects that could relate to each other.  
> Beyond that we've extended the heck out of it, and probably not  
> canonically, although in our defense Class::DBI is only designed to  
> have a canonical usage for the theoretical database model, not a  
> whole application.
>
> The new version of Connotea Code has a Bibliotech::Query with even  
> more speed optimizations but again it's another layer of complexity.
>
>> [SOLUTION:  implied by the last comment DBIx::Class (DBIC)  is  
>> the  natural successor to CDBI, originally developed by the author  
>> of  CDBI::Sweet.  DBIC + memcached has production users.   
>> Discussion on  IRC  (#dbix-class on irc.perl.org) indicates to me  
>> that swappable  caching engines (including a null cache for  
>> debugging) ) should be  trivial and transparent with a DBIC based  
>> data model.  The killer  dbic feature for me is that running your  
>> dbic script like so:   "DBIC_TRACE=1 perl run_my_script.pl" gives  
>> you the exact sql being  passed to the dbic classes.  This outputs  
>> to standard error, thus  making debugging hugely easier.  As  
>> noted, there's a CDBICompat layer  to ease transition from CDBI to  
>> DBIC.  CDBICompat has a more  extensive test suite than CDBI for  
>> good measure.  Finally DBIC tends  to result in much more efficent  
>> SQL than CDBI and swapping out  database engines (eg mysql to pg  
>> to sqlite) is much easier - I've  seen this done trivially from pg  
>> to sqlite for a database  representing a directed acyclic graph. ]
>
> I've got SQL tracing - everything except do() calls and without  
> question-mark translation - if you call Bibliotech::DBI- 
> >activate_warn_sql(). In mod_perl this sends up in Apache's error_log.
>
> Again I've hard of DBIx::Class but was not aware that it offered  
> any major benefits.
>
>> 3.  Template toolkit.  The templates in connotea seem to be  
>> populated  with weird coderefs that make debugging/interrogation  
>> even more  difficult.  TT is one of the best templating solutions  
>> out there, but  connetea seems to misuse it. [SOLUTION:  Port to  
>> explicit templates  resident in files, using PROCESS, INCLUDE and  
>> MACRO blocks where  appropriate.  This should be fairly simple to  
>> do concurrently with  the rest of the catalyst/dbic port. ]
>
> Anything that looks like misuse of TT is probably a result of  
> yielding functionality from our original Component system to TT,  
> where the rest of the Component system is still there, and which  
> was more capable than what we perceived TT could do, and we needed  
> to preserve functionality.
>
> It may be that TT could do more than we realized, or could not then  
> but can now, and in that case it would be, as you point out, better  
> to use the native TT functionality rather than homegrown  
> functionality. I agree on principle.
>
> As an example, our Component system was designed to put all the  
> logic for a component in one place, and avoid expensive  
> recalculation. To that end, a component's result is not just a  
> snippet of HTML like an INCLUDE, but an object that contains HTML  
> parts that can be put in different places on the calling web page,  
> as well as the Javascript for the <head> and Javascript for <body  
> onload=""> that facilitate the component, so if you want to, say,  
> position the cursor in the first form field, you can have the  
> component return the HTML form and that Javascript together.
>
> The new Connotea Code release has a section in the README to cover  
> the TT function calls which should explain things better.
>
>> 4.  Connotea doesn't scale down well.  I've used it on an ibook  
>> g4  for testing (performance verging on reasonable), and a pentium  
>> 3  linux machine with 128mb of RAM where performance was  
>> unacceptable.  [SOLUTON:  port to catalyst where fast_cgi,  
>> mod_perl and other more  exotic engines are all viable solutions.   
>> Removing the mod_perl  depencency opens up shared hosting  
>> possibilities]
>
> We used to have a bibliotech.cgi that could do exactly that (using  
> Apache::Emulator) but we dropped it because NPG had no need for it  
> and it was after all just a hack. I'd be happy to share it with you  
> if you are interested, but the added memcached requirement is the  
> main obstacle I think.
>
>> There are some excellent things about connotea (auto-import and  
>> the  database schema being two big examples), but the above are  
>> show  stoppers which are going to cause huge problems for the   
>> sustainability of the project.  The usability for end users is  
>> great,  but as far as programmer-usability goes connotea needs  
>> major  improvements otherwise it doesn't have a future as a viable  
>> open  source project.  Personally I'd like to hack in storage of  
>> pdf/other  fulltext into the database, but I can't do this in it's  
>> current  state.  I can also see why NPG won't do this themselves.
>
> Thanks for the compliments!
>
> At least you're not complaining it's not Ruby on Rails. ;-)
>
> That's a joke, but I acknowledge that CPAN marches forwards even  
> after we have selected our libraries and yes platforms make a huge  
> difference. We selected solutions to match the engineering and then  
> we tried to move to some newer approaches as dictated by practical  
> concerns. Even moving to TT was only a direct response to needing  
> non-programmers to do editing and design work.
>
>> My vision is for distributed collaborative bibliographies, which  
>> is  why I'd like so much for the project to scale down to a level  
>> where  it could be used by 2-10 researchers on shared hosting.  I  
>> think  optional  re-import back to a master connotea  would be  
>> fairly easy  to implement after the scaling down problem was  
>> addressed.
>
> Your work sounds interesting.
>
>> Part of the problem is that connotea came about at a time of  
>> great  flux during the web app programming space, and the  
>> technology to  program these things has improved massively during  
>> this time.
>
> You are saying something that sounds to me like:
>
> - refactoring for better library support would make easier development
>
> I'd also add:
>
> - like all things, more man-hours on the codebase for general  
> refactoring or more test suite scripts would make easier  
> development, keeping the same libraries even
>
> - refactoring to remove some early assumptions would make a cleaner  
> codebase as well
>
> ...and I'd argue that those two points are just as important.
>
>> Unfortunately it leaves connotea with more technical debt.
>
> Well, I think you may be being a bit hard on us here. ;-)
>
> This is a classic challenge facing companies with working code;  
> whether or not to spend time refactoring things.
>
> When allocating sparse resources carefully, while being customer  
> driven, a lot of the push is inevitably on keeping it running and  
> adding features rather than revising the abstractions for something  
> that already works.
>
> Further, things that may make programming easier but don't actually  
> lower the complexity level of the codebase are not as appealing in  
> general as refactoring projects; e.g. switching from Class::DBI to  
> DBIx::Class is not going to make it easier to understand, (maybe)  
> just easier to work with. The concepts don't change much, you still  
> have an abstraction layer that provides SQL support. You and I can  
> appreciate that, but companies generally are interested in  
> refactoring code when it can introduce junior programmers, not just  
> make it easier for the senior programmers.
>
> Having said that, your suggestions that we may be doing something  
> in a non-standard manner, if they can be done in a standard manner,  
> I do consider bugs to fix, as we should do things the same way as  
> everyone else where possible.
>
> If Class::DBI -> DBIx::Class is thought of as that type of problem,  
> then I can appreciate the desire to switch over. But I'm speaking  
> here more of things like using homegrown approaches over TT  
> directives, as discussed above.
>
>> I'm happy to clarify anything I've written here on request.  If  
>> the  response is going to be "isn't going to happen, sorry about  
>> that",  I'm prepared to accept that too.  As a first step I'd  
>> recommend  popping on to #catalyst at irc.perl.org and asking  
>> about porting  mod_perl apps to Catalyst.
>
> I'm not the final word, and even if I was, what you've presented  
> deserves study and thought before issuing the final word. ;-)
>
> I suggest we keep the conversation going.
>
> Regards,
> Martin Flack
>

Re: [Connotea-code-devel] feedback

[Martin F. <ma...@ne...>]

Hi Kieren,

Thanks for your email. I'm responsible for most of the codebase so I 
will step into the firing line first. ;-)

Kieren Diment wrote:
> I've been wanting to write this email for a while, and you can  
> consider me a strong supporter of what Connotea/NPG  is trying to  
> do.  However there are some major problems that I think need to be  
> addressed if connotea is to become a self sustaining open source  
> project with a sustainable developer community.

Again thanks. I'm aware of most of what you raise, and in fact have 
discussed them with folks, but it doesn't hurt to acknowledge on the 
devel list.

Actually this works out nicely since we are about to release a new 
version of Connotea Code literally any day now, and we are committed to 
posting a public darcs repository and inviting more public support. I'm 
actually thinking you have seen the new code by some of your comments.

> A bit about me:  I'm a social researcher and perl hacker.  Most of  
> the perl I write is for research data management, but I do  
> bibliographic stuff, web robots and occasional web applications  
> programming too.  I've been a significant contributor to the Catalyst  
> web application framework (mainly with the documentation and example  
> code).  I'm computer literate, and I've used mod_perl enough to have  
> a good idea of what irritates me about it.  I think that Catalyst is  
> the natural successor for 95% of the mod_perl or Apache::Registry  
> scripts out there.

I've looked at Catalyst but I've not used it.

> 1.  No test suite.  Lack of unit/behaviour tests along with the  
> impoverished debugging environment with mod_perl make for painful  
> development. [ SOLUTION:  Port to catalyst with built in testing  
> utilities and trivial support for perl -d ]

The new version of Connotea Code sports a fledgling test suite. It 
doesn't shoot for code coverage, just to test the basic functions. It's 
an area I'd like to strengthen.

I concur on the mod_perl debugging situation.

> 2.  Class::DBI + memcached.  These seem to be very tightly coupled.   
> Class::DBI is flawed software, and while it has proved very useful,  
> it clearly has severe limitations.  One example is the difficulty of  
> inspecting the sql that's it's generating.  It's main problem is a  
> general lack of transparency and implicitly generated code.  CDBI   
> "died" as an ongoing open source project about a year ago.  The  
> original author of CDBI's last project resulted in him fixing  
> DBIx::Class' CDBI Compatibility layer  so he could port his client's  
> code from CDBI  to DBIx::Class.

Accurate observation. At the time it seemed like the best way to provide 
abstractions that made the model possible.

I'd add a point that you didn't make, which is that Bibliotech::Query is 
  very complicated, not very clear, and the queries it generates are 
huge, but to some extent it's by necessity.

For the type of thing like Connotea, Class::DBI is way too slow left to 
its own devices - a page was taking 800 queries to pull up all the 
pieces of data, and we solved that with a scary-looking but 
fast-executing query. Another problem might be that it was designed to 
be a very flexible query engine, and as it turns out that flexibility is 
obscuring some of the functionality, i.e. tightening the restraints 
might make prettier source code.

What we got from Class::DBI was a convenient abstraction that database 
rows would be objects that could relate to each other. Beyond that we've 
extended the heck out of it, and probably not canonically, although in 
our defense Class::DBI is only designed to have a canonical usage for 
the theoretical database model, not a whole application.

The new version of Connotea Code has a Bibliotech::Query with even more 
speed optimizations but again it's another layer of complexity.

> [SOLUTION:  implied by the last comment DBIx::Class (DBIC)  is the  
> natural successor to CDBI, originally developed by the author of  
> CDBI::Sweet.  DBIC + memcached has production users.  Discussion on  
> IRC  (#dbix-class on irc.perl.org) indicates to me that swappable  
> caching engines (including a null cache for debugging) ) should be  
> trivial and transparent with a DBIC based data model.  The killer  
> dbic feature for me is that running your dbic script like so:   
> "DBIC_TRACE=1 perl run_my_script.pl" gives you the exact sql being  
> passed to the dbic classes.  This outputs to standard error, thus  
> making debugging hugely easier.  As noted, there's a CDBICompat layer  
> to ease transition from CDBI to DBIC.  CDBICompat has a more  
> extensive test suite than CDBI for good measure.  Finally DBIC tends  
> to result in much more efficent SQL than CDBI and swapping out  
> database engines (eg mysql to pg to sqlite) is much easier - I've  
> seen this done trivially from pg to sqlite for a database  
> representing a directed acyclic graph. ]

I've got SQL tracing - everything except do() calls and without 
question-mark translation - if you call 
Bibliotech::DBI->activate_warn_sql(). In mod_perl this sends up in 
Apache's error_log.

Again I've hard of DBIx::Class but was not aware that it offered any 
major benefits.

> 3.  Template toolkit.  The templates in connotea seem to be populated  
> with weird coderefs that make debugging/interrogation even more  
> difficult.  TT is one of the best templating solutions out there, but  
> connetea seems to misuse it. [SOLUTION:  Port to explicit templates  
> resident in files, using PROCESS, INCLUDE and MACRO blocks where  
> appropriate.  This should be fairly simple to do concurrently with  
> the rest of the catalyst/dbic port. ]

Anything that looks like misuse of TT is probably a result of yielding 
functionality from our original Component system to TT, where the rest 
of the Component system is still there, and which was more capable than 
what we perceived TT could do, and we needed to preserve functionality.

It may be that TT could do more than we realized, or could not then but 
can now, and in that case it would be, as you point out, better to use 
the native TT functionality rather than homegrown functionality. I agree 
on principle.

As an example, our Component system was designed to put all the logic 
for a component in one place, and avoid expensive recalculation. To that 
end, a component's result is not just a snippet of HTML like an INCLUDE, 
but an object that contains HTML parts that can be put in different 
places on the calling web page, as well as the Javascript for the <head> 
and Javascript for <body onload=""> that facilitate the component, so if 
you want to, say, position the cursor in the first form field, you can 
have the component return the HTML form and that Javascript together.

The new Connotea Code release has a section in the README to cover the 
TT function calls which should explain things better.

> 4.  Connotea doesn't scale down well.  I've used it on an ibook g4  
> for testing (performance verging on reasonable), and a pentium 3  
> linux machine with 128mb of RAM where performance was unacceptable.  
> [SOLUTON:  port to catalyst where fast_cgi, mod_perl and other more  
> exotic engines are all viable solutions.  Removing the mod_perl  
> depencency opens up shared hosting possibilities]

We used to have a bibliotech.cgi that could do exactly that (using 
Apache::Emulator) but we dropped it because NPG had no need for it and 
it was after all just a hack. I'd be happy to share it with you if you 
are interested, but the added memcached requirement is the main obstacle 
I think.

> There are some excellent things about connotea (auto-import and the  
> database schema being two big examples), but the above are show  
> stoppers which are going to cause huge problems for the  
> sustainability of the project.  The usability for end users is great,  
> but as far as programmer-usability goes connotea needs major  
> improvements otherwise it doesn't have a future as a viable open  
> source project.  Personally I'd like to hack in storage of pdf/other  
> fulltext into the database, but I can't do this in it's current  
> state.  I can also see why NPG won't do this themselves.

Thanks for the compliments!

At least you're not complaining it's not Ruby on Rails. ;-)

That's a joke, but I acknowledge that CPAN marches forwards even after 
we have selected our libraries and yes platforms make a huge difference. 
We selected solutions to match the engineering and then we tried to move 
to some newer approaches as dictated by practical concerns. Even moving 
to TT was only a direct response to needing non-programmers to do 
editing and design work.

> My vision is for distributed collaborative bibliographies, which is  
> why I'd like so much for the project to scale down to a level where  
> it could be used by 2-10 researchers on shared hosting.  I think  
> optional  re-import back to a master connotea  would be fairly easy  
> to implement after the scaling down problem was addressed.

Your work sounds interesting.

> Part of the problem is that connotea came about at a time of great  
> flux during the web app programming space, and the technology to  
> program these things has improved massively during this time.   

You are saying something that sounds to me like:

- refactoring for better library support would make easier development

I'd also add:

- like all things, more man-hours on the codebase for general 
refactoring or more test suite scripts would make easier development, 
keeping the same libraries even

- refactoring to remove some early assumptions would make a cleaner 
codebase as well

...and I'd argue that those two points are just as important.

> Unfortunately it leaves connotea with more technical debt.

Well, I think you may be being a bit hard on us here. ;-)

This is a classic challenge facing companies with working code; whether 
or not to spend time refactoring things.

When allocating sparse resources carefully, while being customer driven, 
a lot of the push is inevitably on keeping it running and adding 
features rather than revising the abstractions for something that 
already works.

Further, things that may make programming easier but don't actually 
lower the complexity level of the codebase are not as appealing in 
general as refactoring projects; e.g. switching from Class::DBI to 
DBIx::Class is not going to make it easier to understand, (maybe) just 
easier to work with. The concepts don't change much, you still have an 
abstraction layer that provides SQL support. You and I can appreciate 
that, but companies generally are interested in refactoring code when it 
can introduce junior programmers, not just make it easier for the senior 
programmers.

Having said that, your suggestions that we may be doing something in a 
non-standard manner, if they can be done in a standard manner, I do 
consider bugs to fix, as we should do things the same way as everyone 
else where possible.

If Class::DBI -> DBIx::Class is thought of as that type of problem, then 
I can appreciate the desire to switch over. But I'm speaking here more 
of things like using homegrown approaches over TT directives, as 
discussed above.

> I'm happy to clarify anything I've written here on request.  If the  
> response is going to be "isn't going to happen, sorry about that",  
> I'm prepared to accept that too.  As a first step I'd recommend  
> popping on to #catalyst at irc.perl.org and asking about porting  
> mod_perl apps to Catalyst.

I'm not the final word, and even if I was, what you've presented 
deserves study and thought before issuing the final word. ;-)

I suggest we keep the conversation going.

Regards,
Martin Flack

[Connotea-code-devel] feedback

[Kieren D. <ki...@di...>]

Hi,

I've been wanting to write this email for a while, and you can  
consider me a strong supporter of what Connotea/NPG  is trying to  
do.  However there are some major problems that I think need to be  
addressed if connotea is to become a self sustaining open source  
project with a sustainable developer community.

A bit about me:  I'm a social researcher and perl hacker.  Most of  
the perl I write is for research data management, but I do  
bibliographic stuff, web robots and occasional web applications  
programming too.  I've been a significant contributor to the Catalyst  
web application framework (mainly with the documentation and example  
code).  I'm computer literate, and I've used mod_perl enough to have  
a good idea of what irritates me about it.  I think that Catalyst is  
the natural successor for 95% of the mod_perl or Apache::Registry  
scripts out there.

Problems I've experienced with connotea as an open source project.   
Suggested solutions in [square brackets].  Roughly in order of  
importance.

1.  No test suite.  Lack of unit/behaviour tests along with the  
impoverished debugging environment with mod_perl make for painful  
development. [ SOLUTION:  Port to catalyst with built in testing  
utilities and trivial support for perl -d ]

2.  Class::DBI + memcached.  These seem to be very tightly coupled.   
Class::DBI is flawed software, and while it has proved very useful,  
it clearly has severe limitations.  One example is the difficulty of  
inspecting the sql that's it's generating.  It's main problem is a  
general lack of transparency and implicitly generated code.  CDBI   
"died" as an ongoing open source project about a year ago.  The  
original author of CDBI's last project resulted in him fixing  
DBIx::Class' CDBI Compatibility layer  so he could port his client's  
code from CDBI  to DBIx::Class.

[SOLUTION:  implied by the last comment DBIx::Class (DBIC)  is the  
natural successor to CDBI, originally developed by the author of  
CDBI::Sweet.  DBIC + memcached has production users.  Discussion on  
IRC  (#dbix-class on irc.perl.org) indicates to me that swappable  
caching engines (including a null cache for debugging) ) should be  
trivial and transparent with a DBIC based data model.  The killer  
dbic feature for me is that running your dbic script like so:   
"DBIC_TRACE=1 perl run_my_script.pl" gives you the exact sql being  
passed to the dbic classes.  This outputs to standard error, thus  
making debugging hugely easier.  As noted, there's a CDBICompat layer  
to ease transition from CDBI to DBIC.  CDBICompat has a more  
extensive test suite than CDBI for good measure.  Finally DBIC tends  
to result in much more efficent SQL than CDBI and swapping out  
database engines (eg mysql to pg to sqlite) is much easier - I've  
seen this done trivially from pg to sqlite for a database  
representing a directed acyclic graph. ]

3.  Template toolkit.  The templates in connotea seem to be populated  
with weird coderefs that make debugging/interrogation even more  
difficult.  TT is one of the best templating solutions out there, but  
connetea seems to misuse it. [SOLUTION:  Port to explicit templates  
resident in files, using PROCESS, INCLUDE and MACRO blocks where  
appropriate.  This should be fairly simple to do concurrently with  
the rest of the catalyst/dbic port. ]

4.  Connotea doesn't scale down well.  I've used it on an ibook g4  
for testing (performance verging on reasonable), and a pentium 3  
linux machine with 128mb of RAM where performance was unacceptable.  
[SOLUTON:  port to catalyst where fast_cgi, mod_perl and other more  
exotic engines are all viable solutions.  Removing the mod_perl  
depencency opens up shared hosting possibilities]

There are some excellent things about connotea (auto-import and the  
database schema being two big examples), but the above are show  
stoppers which are going to cause huge problems for the  
sustainability of the project.  The usability for end users is great,  
but as far as programmer-usability goes connotea needs major  
improvements otherwise it doesn't have a future as a viable open  
source project.  Personally I'd like to hack in storage of pdf/other  
fulltext into the database, but I can't do this in it's current  
state.  I can also see why NPG won't do this themselves.

My vision is for distributed collaborative bibliographies, which is  
why I'd like so much for the project to scale down to a level where  
it could be used by 2-10 researchers on shared hosting.  I think  
optional  re-import back to a master connotea  would be fairly easy  
to implement after the scaling down problem was addressed.

Part of the problem is that connotea came about at a time of great  
flux during the web app programming space, and the technology to  
program these things has improved massively during this time.   
Unfortunately it leaves connotea with more technical debt.

I'm happy to clarify anything I've written here on request.  If the  
response is going to be "isn't going to happen, sorry about that",  
I'm prepared to accept that too.  As a first step I'd recommend  
popping on to #catalyst at irc.perl.org and asking about porting  
mod_perl apps to Catalyst.

--
Kieren Diment
Centre for Leadership and Knowledge Management
School of Management, Marketing

[Connotea-code-devel] <ADV>Affordable Quality IT And Design Services

[<sa...@on...>]

Dear connotea-code-devel,=09
 =09
    Are you looking for IT & design freelancers? We, ONGGIE provide full
range of affordable quality IT & design services to meet your interest:

	=C2=BB Application Development
=C2=BB Website Design
=C2=BB Photo-editing / Wedding Album Design
=C2=BB Corporate Stationary Design
=C2=BB Graphic Design
=C2=BB Multimedia Presentation
=C2=BB Application Interface Design
=C2=BB Painting=09
 =09
    We found your contact from lists.sourceforge.net. We believe our
service has the capability to expand and strengthen your company's
potential by improving the IT workflow and corporate presentation. We
have experienced expertise in software development and creative designs
to ensure the achievements of your investments. Your IT requirements
will be addressed with the utmost concern while our professionals
undertake every care to deliver beyond your expectations.=09
 =09
    For further information, please call: +65 68260618 or visit
http://www.onggie.com <http://www.onggie.com/>  for a glance of our
creative works.=09
 =09
 =09
    To unsubscribe from this mailing list, please reply to this message.

 =09
Best Regards,=09
ONGGIE=09

[Connotea-code-devel] connotea java api client

[Christopher T. <chr...@ch...>]

Hi all,

I've packaged up an early beta of a Java client for the Connotea API.
There are still a few outstanding "todos" (e.g. write a proper unit test
for the main client class ;D) but it is basically ready and comments
from anyone who wants to try it out now would be very welcome.

At present, there is only a pre-compiled jar and javadocs. The source
etc will be made available as soon as we have found a permanent home and
determined what licence the code should be released under etc etc.

Temporary homes for both of the above:

jar file: http://nascent.nature.com/connotea/connotea.jar
api docs: http://nascent.nature.com/connotea/api/

Please do try it out and report back any bugs you encounter to this
list, or directly to me (via author link in api docs).

Likewise, if there are any glaring omissions or misrepresentations of
the Connotea API, just let me know.

Cheers,

Chris

Re: [Connotea-code-devel] post not implemented ?

[Martin F. <ma...@ne...>]

Thanks Chris, looking forward to the Java API.

Christopher Townson wrote:
> p.s. has anyone else noticed that if you pass an invalid (i.e.
> unencoded) url to /data/remove you get a default server 404, whereas
> doing the same with any of the gets returns an RDF response? Seems like
> one to fix?

Interesting. The code doesn't specify that to happen so it must be a 
mod_perl issue. I'll put it on the to-do list. Thanks.

Martin

[Connotea-code-devel] investor_alert-47312

[Tony <Ton...@ha...>]

[Connotea-code-devel] Pictures

[Fanny W. <a-a...@ad...>]

Hello! I am tired this afternoon. I am nice girl that would like to chat with you. Email me at ky...@li... only, because I am writing not from my personal email. Will send some of my pictures

Re: [Connotea-code-devel] post not implemented ?

[Christopher T. <chr...@ch...>]

Benjamin Good wrote:
> If anyone else is having trouble reading and writing to the api from  
> java, please do let me know - think I've made it through most of the  
> cases.  The only one I haven't finished yet is the job of writing to  
> the connotea wiki from code.  Seems that it is possible to do that  
> via browser emulation (cookies etc.) but it would be cool if it was  
> added to the API.
> 

Hi Ben,

(To introduce myself: I'm a developer in the Emerging Technologies
Department at Nature.)

Just so you know, I'm just in the process of wrapping up a Java Connotea
API client. I haven't encountered any major problems implementing this
yet (fingers crossed ;D) ... there are a few small things that I was
going to summarize on this list as soon as I had wrapped the thing up,
but they were mainly questions to do with things like  "Why not HTTP
delete for remove?" etc.

Posting does work - my client is built on Restlet
(http://www.restlet.org/), which I can highly recommend and it will make
your task a million times simpler than using HttpURLConnection and
friends directly.

Here is a sample piece of code I wrote during development which works
for me and posts a link to Connotea via the API:


package org.connotea;

import org.restlet.Client;
import org.restlet.data.ChallengeResponse;
import org.restlet.data.ChallengeScheme;
import org.restlet.data.Form;
import org.restlet.data.Method;
import org.restlet.data.Parameter;
import org.restlet.data.Protocol;
import org.restlet.data.Reference;
import org.restlet.data.Request;
import org.restlet.data.Response;

public class Driver
{
    public static void main(String[] args) throws Throwable
    {
        Client client = new Client(Protocol.HTTP);

        Reference reference = new
Reference("http://www.connotea.org/data/add");

        Request request = new Request(Method.POST, reference);

        ChallengeScheme challengeScheme = ChallengeScheme.HTTP_BASIC;
        ChallengeResponse challengeResponse = new ChallengeResponse(
                challengeScheme, "your_username", "your_password");
        request.setChallengeResponse(challengeResponse);

        Form form = new Form();
        form.add(new Parameter("uri", "http://www.nature.com/"));
        form.add(new Parameter("tags", "science journal"));
        form.encode();

        request.setEntity(form.getWebRepresentation());

        Response response = client.handle(request);
        System.out.println(response.getStatus());
    }
}

I hope to complete a beta of the Java client I am working on today, as
it happens, so you could just wait for that if you like. The above code
demonstrates how Java client is really working "under the hood", but it
provides a front-end and Connotea "domain objects" which encapsulate all
the currently implemented API methods for convenient use in your
applications.

All the best,

Chris

p.s. has anyone else noticed that if you pass an invalid (i.e.
unencoded) url to /data/remove you get a default server 404, whereas
doing the same with any of the gets returns an RDF response? Seems like
one to fix?

Re: [Connotea-code-devel] post not implemented ?

[Benjamin G. <go...@in...>]

OK,

Apologies all, for using this list as my own personal "intro to java  
web programming" course..  Just to finish this off, the following  
code frag does what I was trying to do

String encodedData = "uri="+URLEncoder.encode(uri,"UTF-8") 
+"&tags="+URLEncoder.encode(tags,"UTF-8");		
URL connoU = new URL("http://www.connotea.org/data/add");			
HttpURLConnection conno = (HttpURLConnection) connoU.openConnection();	
conno.setDoOutput(true);
OutputStream os = conno.getOutputStream();
os.write( encodedData.getBytes() );
int code = conno.getResponseCode();
System.out.println("response code from connotea="+code);

If anyone else is having trouble reading and writing to the api from  
java, please do let me know - think I've made it through most of the  
cases.  The only one I haven't finished yet is the job of writing to  
the connotea wiki from code.  Seems that it is possible to do that  
via browser emulation (cookies etc.) but it would be cool if it was  
added to the API.

cheers all
-Ben



On Aug 2, 2007, at 5:45 PM, Martin Flack wrote:

> I don't know Java libraries, but it seems unlikely that you should  
> be using setRequestProperty() for both User-Agent (an HTTP header)  
> and uri (a form parameter).
>
> POST's are supposed to put their parameters in the payload (like  
> your web browser does) by spec.
>
> Take a look at how they encode and attach the form payload on these  
> examples:
> http://www.javaworld.com/javaworld/javatips/jw-javatip34.html
> http://developers.sun.com/mobility/midp/ttips/HTTPPost/
>
> Cheers,
> Martin
>
> Benjamin Good wrote:
>> Usually adding the parameters into the URL works .. why would it  
>> not in this case?
>> I get the same error (400) if I set the params in the code
>> URL connoU = new URL("http://www.connotea.org/data/add");
>> HttpURLConnection conno = (HttpURLConnection) connoU.openConnection 
>> ();
>> conno.setRequestProperty("User-Agent", "WWW::Connotea");
>> conno.setRequestProperty("uri", URLEncoder.encode(uri,"UTF-8"));
>> conno.setRequestProperty("tag", URLEncoder.encode(tag,"UTF-8"));
>> conno.setRequestMethod("POST");
>> I've been digging for some else that has this running in Java, but  
>> the only one I could find (from Pierre Lindenbaum) didn't use the  
>> web api - it emulated a user session by storing and sending  
>> cookies for authentication and sending requests including button  
>> clicks.  (His ConnoteaClient program - but it doesn't seem to be  
>> work ing anymore).
>> -Ben
>

Re: [Connotea-code-devel] post not implemented ?

[Benjamin G. <go...@in...>]

Yes..  that did seem strange and is, of course, wrong.  Apologies for  
my idiocy.
However, I can't get it to work with the methods you directed me to  
either.  Always get the 400 response.

any other ideas?
-Ben

String uri = "http://www.cnn.com";
String tags = "test123";

String encodedData = URLEncoder.encode("uri="+uri 
+"&tags="+tags,"UTF-8");
		
			URL connoU = new URL("http://www.connotea.org/data/add");			
			HttpURLConnection conno = (HttpURLConnection) connoU.openConnection 
();
			
			conno.setRequestProperty("User-Agent", "WWW::Connotea");
			conno.setRequestProperty("Content-Type", "application/x-www-form- 
urlencoded");
			conno.setRequestMethod("POST");
			conno.setRequestProperty( "Content-Length", ""+encodedData.length 
() );
			conno.setDoOutput(true);
			conno.setDoInput(true);
			conno.setUseCaches(false);
			OutputStream os = conno.getOutputStream();
			os.write( encodedData.getBytes() );
			os.flush();
			os.close();
			int code = conno.getResponseCode();
			System.out.println("response code from connotea="+code);



On Aug 2, 2007, at 5:45 PM, Martin Flack wrote:

> I don't know Java libraries, but it seems unlikely that you should  
> be using setRequestProperty() for both User-Agent (an HTTP header)  
> and uri (a form parameter).
>
> POST's are supposed to put their parameters in the payload (like  
> your web browser does) by spec.
>
> Take a look at how they encode and attach the form payload on these  
> examples:
> http://www.javaworld.com/javaworld/javatips/jw-javatip34.html
> http://developers.sun.com/mobility/midp/ttips/HTTPPost/
>
> Cheers,
> Martin
>
> Benjamin Good wrote:
>> Usually adding the parameters into the URL works .. why would it  
>> not in this case?
>> I get the same error (400) if I set the params in the code
>> URL connoU = new URL("http://www.connotea.org/data/add");
>> HttpURLConnection conno = (HttpURLConnection) connoU.openConnection 
>> ();
>> conno.setRequestProperty("User-Agent", "WWW::Connotea");
>> conno.setRequestProperty("uri", URLEncoder.encode(uri,"UTF-8"));
>> conno.setRequestProperty("tag", URLEncoder.encode(tag,"UTF-8"));
>> conno.setRequestMethod("POST");
>> I've been digging for some else that has this running in Java, but  
>> the only one I could find (from Pierre Lindenbaum) didn't use the  
>> web api - it emulated a user session by storing and sending  
>> cookies for authentication and sending requests including button  
>> clicks.  (His ConnoteaClient program - but it doesn't seem to be  
>> work ing anymore).
>> -Ben
>

Re: [Connotea-code-devel] post not implemented ?

[Martin F. <ma...@ne...>]

I don't know Java libraries, but it seems unlikely that you should be 
using setRequestProperty() for both User-Agent (an HTTP header) and uri 
(a form parameter).

POST's are supposed to put their parameters in the payload (like your 
web browser does) by spec.

Take a look at how they encode and attach the form payload on these 
examples:
http://www.javaworld.com/javaworld/javatips/jw-javatip34.html
http://developers.sun.com/mobility/midp/ttips/HTTPPost/

Cheers,
Martin

Benjamin Good wrote:
> Usually adding the parameters into the URL works .. why would it not in 
> this case?
> 
> I get the same error (400) if I set the params in the code
> 
> URL connoU = new URL("http://www.connotea.org/data/add");
> 
> 
> HttpURLConnection conno = (HttpURLConnection) connoU.openConnection();
> conno.setRequestProperty("User-Agent", "WWW::Connotea");
> conno.setRequestProperty("uri", URLEncoder.encode(uri,"UTF-8"));
> conno.setRequestProperty("tag", URLEncoder.encode(tag,"UTF-8"));
> conno.setRequestMethod("POST");
> 
> I've been digging for some else that has this running in Java, but the 
> only one I could find (from Pierre Lindenbaum) didn't use the web api - 
> it emulated a user session by storing and sending cookies for 
> authentication and sending requests including button clicks.  (His 
> ConnoteaClient program - but it doesn't seem to be work ing anymore).
> 
> -Ben

Re: [Connotea-code-devel] post not implemented ?

[Benjamin G. <go...@in...>]

Usually adding the parameters into the URL works .. why would it not  
in this case?

I get the same error (400) if I set the params in the code

	URL connoU = new URL("http://www.connotea.org/data/add");
			
	HttpURLConnection conno = (HttpURLConnection) connoU.openConnection();
	conno.setRequestProperty("User-Agent", "WWW::Connotea");
	conno.setRequestProperty("uri", URLEncoder.encode(uri,"UTF-8"));
	conno.setRequestProperty("tag", URLEncoder.encode(tag,"UTF-8"));
	conno.setRequestMethod("POST");

I've been digging for some else that has this running in Java, but  
the only one I could find (from Pierre Lindenbaum) didn't use the web  
api - it emulated a user session by storing and sending cookies for  
authentication and sending requests including button clicks.  (His  
ConnoteaClient program - but it doesn't seem to be work ing anymore).

-Ben


On Aug 2, 2007, at 4:57 PM, Martin Flack wrote:

> Your example does not seem to set a payload though. The URL should  
> be free of parameters and their should be content that contains the  
> parameters, encoded with application/x-www-form-urlencoded.
>
> Benjamin Good wrote:
>> Had already tried that one (as well as pretending to be a browser).
>> Doing my best to figure out what the difference is.. but no luck  
>> so far.  Here is the method I'm using (authentication not included  
>> but working fine).
>> -ben
>

Re: [Connotea-code-devel] post not implemented ?

[Martin F. <ma...@ne...>]

Your example does not seem to set a payload though. The URL should be 
free of parameters and their should be content that contains the 
parameters, encoded with application/x-www-form-urlencoded.

Benjamin Good wrote:
> Had already tried that one (as well as pretending to be a browser).
> Doing my best to figure out what the difference is.. but no luck so 
> far.  Here is the method I'm using (authentication not included but 
> working fine).
> 
> -ben

Re: [Connotea-code-devel] post not implemented ?

[Benjamin G. <go...@in...>]

Had already tried that one (as well as pretending to be a browser).
Doing my best to figure out what the difference is.. but no luck so  
far.  Here is the method I'm using (authentication not included but  
working fine).

-ben

public String postToConnotea(String url){
		String response = "";
		BufferedReader in = null;
		try {
			URL connoU = new URL(url);

			HttpURLConnection conno = (HttpURLConnection) connoU.openConnection 
();
			conno.setRequestProperty("User-Agent", "WWW::Connotea");
			conno.setRequestMethod("POST");
			conno.setDoOutput(true);
			int code = conno.getResponseCode();
			System.out.println("code "+code+" for request url "+url);
			switch(code){
			case 400:
				conno.disconnect();
				System.err.println("400 - Server did not like form of the request");
				return null;
			case 401:
				conno.disconnect();
				System.err.println("401 - Server did not accept user credentials");
				return null;
			case 404:
				conno.disconnect();
				System.err.println("404 - Query returned no results");
				return null;
			case 503:
				conno.disconnect();
				System.err.println("503 - Connotea server failed to respond");
				return null;
			default: //200 all good
				System.out.println("server accepted request, retrieving data");
				in = new BufferedReader(
						new InputStreamReader(
								conno.getInputStream()));
				if(in!=null){
					response = in.readLine();
					in.close();
				}
				conno.disconnect();
			}

		}catch (MalformedURLException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (Exception e) { //catch other exceptions here (like  
SaxParsing exceptions reading RDF)
			e.printStackTrace();
		}
		return response;
	}




On Aug 2, 2007, at 2:28 PM, Martin Flack wrote:

> Try adding WWW::Connotea in the User-Agent string.
>
> Other than that just try to see what the Java program is doing  
> differently...?
>
> Martin
>
> Benjamin Good wrote:
>> Hi Martin,
>> Switching to POST changes the error to a 400 .
>> I have it working from a simple html form posting through  
>> firefox,  but am trying to get it going in java.  The java program  
>> succeeds in  GETting data from connotea - just can't post.
>> Is there a particular user-agent or other request property that i   
>> should be using?
>> thanks
>> -Ben
>

Re: [Connotea-code-devel] post not implemented ?

[Martin F. <ma...@ne...>]

Try adding WWW::Connotea in the User-Agent string.

Other than that just try to see what the Java program is doing 
differently...?

Martin

Benjamin Good wrote:
> Hi Martin,
> 
> Switching to POST changes the error to a 400 .
> 
> I have it working from a simple html form posting through firefox,  
> but am trying to get it going in java.  The java program succeeds in  
> GETting data from connotea - just can't post.
> Is there a particular user-agent or other request property that i  
> should be using?
> 
> thanks
> -Ben

Re: [Connotea-code-devel] post not implemented ?

[Benjamin G. <go...@in...>]

Hi Martin,

Switching to POST changes the error to a 400 .

I have it working from a simple html form posting through firefox, =20
but am trying to get it going in java.  The java program succeeds in =20
GETting data from connotea - just can't post.
Is there a particular user-agent or other request property that i =20
should be using?

thanks
-Ben


On Aug 2, 2007, at 12:34 PM, Martin Flack wrote:

> Ben,
>
> Has to be HTTP POST not GET.
>
> I'm attaching a script that I developed for testing.
>
> Martin
>
> Benjamin Good wrote:
>> Hi there,
>>   Is there anything wrong with this ?
>> http://www.connotea.org/data/add?uri=3Dhttp%3A%2F% =20
>> 2Fbioinfo.icapture.ubc.ca%2Fekawas&tags=3Dtest%2C+%22test+post%22 =20
>> +another+test
>> I think authentication is working fine, but I get a 501 response
>> =E2=88=92
>> 	<rdf:RDF>
>> =E2=88=92
>> 	<Response rdf:about=3D"">
>> <code>501</code>
>> <message>Not Implemented</message>
>> <isFailure>1</isFailure>
>> <user>MrED</user>
>> <apiVersion>0.1.0</apiVersion>
>> <bibliotechVersion>1.7.1</bibliotechVersion>
>> </Response>
>> </rdf:RDF>
>> whats up ?
>> thanks
>> -Ben
>
> #!/bin/bash
> # post something to Connotea from the command line, requires lwp-=20
> request
> # lwp-request is part of libwww-perl which is a package on some =20
> Linux platforms and available on CPAN
> # edit the variables below for input
> # output is a dump of the HTTP transaction to stdout including =20
> Connotea's XML response
>
> user=3DEDIT
> pass=3DEDIT
> host=3Dwww.connotea.org
> # note all following fields should be url escaped
> uri=3Dhttp%3A//www.cnn.com
> # separate tags with %20 (spacebar)
> tags=3Dabc%20def%20ghi
> # could also pass in usertitle, description, mywork, private, etc.
>
> echo "uri=3D$uri&tags=3D$tags" | lwp-request -C "$user:$pass" -u -U -s =
-=20
> S -e -m POST "http://$host/data/add"
>

Re: [Connotea-code-devel] post not implemented ?

[Martin F. <ma...@ne...>]

Ben,

Has to be HTTP POST not GET.

I'm attaching a script that I developed for testing.

Martin

Benjamin Good wrote:
> Hi there,
> 
>   Is there anything wrong with this ?
> http://www.connotea.org/data/add?uri=http%3A%2F% 
> 2Fbioinfo.icapture.ubc.ca%2Fekawas&tags=test%2C+%22test+post%22 
> +another+test
> 
> I think authentication is working fine, but I get a 501 response
> 
> −
> 	<rdf:RDF>
> −
> 	<Response rdf:about="">
> <code>501</code>
> <message>Not Implemented</message>
> <isFailure>1</isFailure>
> <user>MrED</user>
> <apiVersion>0.1.0</apiVersion>
> <bibliotechVersion>1.7.1</bibliotechVersion>
> </Response>
> </rdf:RDF>
> 
> whats up ?
> 
> thanks
> -Ben

[Connotea-code-devel] post not implemented ?

[Benjamin G. <go...@in...>]

Hi there,

  Is there anything wrong with this ?
http://www.connotea.org/data/add?uri=3Dhttp%3A%2F%=20
2Fbioinfo.icapture.ubc.ca%2Fekawas&tags=3Dtest%2C+%22test+post%22=20
+another+test

I think authentication is working fine, but I get a 501 response

=E2=88=92
	<rdf:RDF>
=E2=88=92
	<Response rdf:about=3D"">
<code>501</code>
<message>Not Implemented</message>
<isFailure>1</isFailure>
<user>MrED</user>
<apiVersion>0.1.0</apiVersion>
<bibliotechVersion>1.7.1</bibliotechVersion>
</Response>
</rdf:RDF>

whats up ?

thanks
-Ben

Re: [Connotea-code-devel] Approaches to solving Buggotea

[Thomas B. <pro...@gm...>]

To me it is all about Shannon.

Our university has a campus subscription to Nature.  When a student  
is on campus, he can link straight through, e.g., to Martin Kemp's  
article,

     http://www.nature.com/nature/journal/v434/n7031/full/434308a.html

but if the student is off campus, he has to go through the campus  
library's proxy server to hit that article, viz.

     http://www.nature.com.ucfproxy.fcla.edu/nature/journal/v434/ 
n7031/full/434308a.html

When a student bookmarks that article from campus, the bookmark is  
distinguishable from the bookmark of a student off campus.   
Distinguishing information persists in the bookmarks, and I would  
vote for allowing bookmarks to retain the distinguishing information  
intact, unless the library owner decides otherwise.

I say all that from thinking about the information content of the  
Connotea system: keep information level high, don't lose information  
carelessly.  The more ways information links together, the higher the  
entropy -- in the Shannon sense.

Also, I am not thinking about ease of use, the "university user" idea  
that Martin mentions.  That is something to think about, but to me it  
is separate from the information content of the Connotea system.

And I do instruct my students about the OpenURL resolver in their  
advanced settings.

- Thomas Brueckner




On Aug 1, 2007, at 2:52 PM, Martin Flack wrote:

> Hi Thomas,
>
> Thomas Brueckner wrote:
>> This is all an issue for me because my students in Connotea are a   
>> mixture of students on campus and students off campus.  Their  
>> links  to Nature for instance, are pure nature.com URIs if on  
>> campus, but  off campus student links carry the university's proxy  
>> information  embedded in the URI.  I would prefer that Connotea  
>> NOT destroy that  distinguishing information.  And perhaps others  
>> might care to analyze  those access methods for Connotea as a whole.
>
> You're speaking of a duality of links that you currently maintain  
> outside of Connotea, right? Or are you accomplishing it in the  
> library with tags or something?
>
> The article-to-many-bookmarks thoughts have made me realize that  
> there may be cases, such as yours here, where a user might  
> *automatically* benefit from more than one URL being available to  
> them. We could call it the "university user issue", where they  
> would like one type or URI on campus and another related type off  
> campus. Although this sounds like something we might do with plugin  
> modules instead of the buggotea solution, because it depends on the  
> given domain being linked.
>
> Martin
>

Re: [Connotea-code-devel] Approaches to solving Buggotea

[Martin F. <ma...@ne...>]

Roderic Page wrote:
> The issue of campus versus off-campus access seems to the problem  
> that OpenURL was designed to solve. Connotea already supports OpenURL  
> (http://www.connotea.org/advanced), so one way to deal this issue if  
> for students to bookmark using DOIs, and to set the OpenURL Resolver  
> for their account to one provided by their host institution (assuming  
> one exists). This means they get an authoritative link (the DOI),  
> plus a "local" link (provided by OpenURL).

You are of course correct that OpenURL is the canonical feature for this.

Martin

Re: [Connotea-code-devel] Approaches to solving Buggotea

[Martin F. <ma...@ne...>]

Hi Thomas,

Thomas Brueckner wrote:
> This is all an issue for me because my students in Connotea are a  
> mixture of students on campus and students off campus.  Their links  
> to Nature for instance, are pure nature.com URIs if on campus, but  
> off campus student links carry the university's proxy information  
> embedded in the URI.  I would prefer that Connotea NOT destroy that  
> distinguishing information.  And perhaps others might care to analyze  
> those access methods for Connotea as a whole.

You're speaking of a duality of links that you currently maintain 
outside of Connotea, right? Or are you accomplishing it in the library 
with tags or something?

The article-to-many-bookmarks thoughts have made me realize that there 
may be cases, such as yours here, where a user might *automatically* 
benefit from more than one URL being available to them. We could call it 
the "university user issue", where they would like one type or URI on 
campus and another related type off campus. Although this sounds like 
something we might do with plugin modules instead of the buggotea 
solution, because it depends on the given domain being linked.

Martin

[Connotea-code-devel] <no subject>

[Mulvany, I. <i.m...@na...>]

From: <mla...@sf...>
Reply-To: <mla...@sf...>
Date: Wed, 01 Aug 2007 08:31:47 -0700
To: <r....@bi...>
Cc: <con...@li...>
Subject: Re: [Connotea-code-devel] Approaches to solving Buggotea

Wow, this sounds great! I didn't realize Connotea had this support.
Hopefully, my institution supports OpenURL.

I agree that linking by DOI seems like the best choice, but I have not been
using this due to the lack of citation information that is captured by
Connotea when using the DOI. I think this problem was brought up recently on
this list, but I am unsure if it is a limit of DOI or just some additional
coding that is needed by Connotea.

Currently, I am always linking to the Pubmed abstract, which can sometimes
be a pain since it makes for an additional step for bookmarking when I am
reading a paper.

If I was not clear the first time, I would prefer if Connotea did not change
the uri I use to bookmark an article, but did let me know that there are
other uri's that refer to the same article.

Morgan


On Wed, 1 Aug 2007 16:09:01 +0100 r....@bi... wrote:
> The issue of campus versus off-campus access seems to the problem
> that OpenURL was designed to solve. Connotea already supports OpenURL
> (http://www.connotea.org/advanced), so one way to deal this issue if
> for students to bookmark using DOIs, and to set the OpenURL Resolver
> for their account to one provided by their host institution (assuming
> one exists). This means they get an authoritative link (the DOI),
> plus a "local" link (provided by OpenURL).
> 
> This would also have the advantage that when and if the students
> leave, their bookmarks that depend on locally resolvable links don't
> suddenly become useless. If they move to an institution like mine
> where access to external resources is by IP address, then they would
> still be able to access papers via the DOI. If their new institution
> supports OpenURL they just have to change the OpenURL resolver.
> 
> HTTP URIs by themselves are very fragile -- linking directly to a
> publisher's page is asking for trouble.
> 
> Regards
> 
> Rod
> 
> 
> 
> On 1 Aug 2007, at 15:49, Thomas Brueckner wrote:
> 
> > I am a user, not yet a developer, and I am not a librarian, but I
> > like the first approach, "THUS, user sees other instances of the
> > article through his/her bookmark," for a few reasons:
> >
> > (1) the URI in the bookmark in my library contains information on my
> > access method to the article;
> > (2) seeing other instances via DOI match would prompt me to clean up
> > my bookmark;
> > (3) seems like it would fit naturally in "Posted by" somewhere, since
> > that array already refers to other libraries.
> >
> > I like keeping information intact (1) but if it can be or ought to be
> > crafted, I want to do it (2).  Morgan L's suggestion is similar to
> > (2).
> >
> > This is all an issue for me because my students in Connotea are a
> > mixture of students on campus and students off campus.  Their links
> > to Nature for instance, are pure nature.com URIs if on campus, but
> > off campus student links carry the university's proxy information
> > embedded in the URI.  I would prefer that Connotea NOT destroy that
> > distinguishing information.  And perhaps others might care to analyze
> > those access methods for Connotea as a whole.
> >
> > - Thomas Brueckner
> > (Univ. of Central Florida)
> >
> >
> > On Aug 1, 2007, at 5:50 AM, Mulvany, Ian wrote:
> >
> >
> >> Hi List,
> >>
> >> We are discussing approaches to solving the long standing issue with
> >> Connotea known colloquially as Buggotea, where the system where the
> >> system
> >> is not checking that two bookmarks might be referring to the same
> >> article if
> >> they have different uri's but the same doi or other identifier.
> >> Martin just
> >> sent me the mail below and I thought I'd send it on to get any
> >> feedback that
> >> you might have.
> >>
> >>
> >>
> >>
> >>> Here's an open philosophical question for you to ponder.
> >>>
> >>> When a user adds a URL, are they merely using it as an accessor to
> >>> present an article that goes into their library, or are they
> >>> linking to
> >>> that URL in particular with a passing interest to where else the
> >>> article
> >>> may exist.
> >>>
> >>> Does that make sense?
> >>>
> >>> It relates to whether we keep the user_bookmark table and add an
> >>> "outer"
> >>> layer of abstraction to group "articles", or whether we break that
> >>> table
> >>> and change it to user_article postings where the articles have
> >>> one or
> >>> more bookmarks.
> >>>
> >>> user_bookmark = one user posts one bookmark (URL)
> >>> article = an abstract idea of an article like a human thinks of it
> >>> article_bookmark = relate abstract article to physical bookmark
> >>> THUS, user sees other instances of the article through his/her
> >>> bookmark
> >>> (slightly easier to engineer on top of what we have now)
> >>>
> >>> ...or....
> >>>
> >>> user_article = one user includes one article
> >>> article = an abstract idea of an article like a human thinks of it
> >>> article_bookmark = relate abstract article to physical bookmark
> >>> THUS, user may be shown different bookmark than originally
> >>> posted, in
> >>> practice we probably wouldn't want to do that often but it could
> >>> be a
> >>> feature, say the user could say "prefer doi resolver today" then
> >>> "prefer
> >>> harvard.edu links today" etc.
> >>> (slightly harder to engineer on top of what we have now)
> >>> (BTW: I believe this is more like CiteULike)
> >>>
> >>> It can work either way, it's just a mental approach. ;-)
> >>>
> >>> Martin
> >>>
> >>
> >> *********************************************************************
> >> *
> >> **********
> >> DISCLAIMER: This e-mail is confidential and should not be used by
> >> anyone who is
> >> not the original intended recipient. If you have received this e-
> >> mail in error
> >> please inform the sender and delete it from your mailbox or any
> >> other storage
> >> mechanism. Neither Macmillan Publishers Limited nor any of its
> >> agents accept
> >> liability for any statements made which are clearly the sender's
> >> own and not
> >> expressly made on behalf of Macmillan Publishers Limited or one of
> >> its agents.
> >> Please note that neither Macmillan Publishers Limited nor any of
> >> its agents
> >> accept any responsibility for viruses that may be contained in this
> >> e-mail or
> >> its attachments and it is your responsibility to scan the e-mail and
> >> attachments (if any). No contracts may be concluded on behalf of
> >> Macmillan
> >> Publishers Limited or its agents by means of e-mail communication.
> >> Macmillan
> >> Publishers Limited Registered in England and Wales with registered
> >> number 785998
> >> Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS
> >> *********************************************************************
> >> *
> >> **********
> >>
> >> ---------------------------------------------------------------------
> >> -
> >> ---
> >> This SF.net email is sponsored by: Splunk Inc.
> >> Still grepping through log files to find problems?  Stop.
> >> Now Search log events and configuration files using AJAX and a
> >> browser.
> >> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> >> _______________________________________________
> >> Connotea-code-devel mailing list
> >> Con...@li...
> >> https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
> >>
> >
> >
> >
> > Dr. Thomas J. Brueckner
> > Dept. of Physics, Univ. of Central Florida
> > 407-823-4541
> > Designing online physics course 2007
> >
> >
> >
> >
> > ----------------------------------------------------------------------
> > ---
> > This SF.net email is sponsored by: Splunk Inc.
> > Still grepping through log files to find problems?  Stop.
> > Now Search log events and configuration files using AJAX and a
> > browser.
> > Download your FREE copy of Splunk now >>  http://get.splunk.com/
> > _______________________________________________
> > Connotea-code-devel mailing list
> > Con...@li...
> > https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
> >
> 
> ----------------------------------------
> Professor Roderic D. M. Page
> Editor, Systematic Biology
> DEEB, IBLS
> Graham Kerr Building
> University of Glasgow
> Glasgow G12 8QP
> United Kingdom
> 
> Phone: +44 141 330 4778
> Fax: +44 141 330 2792
> email: r....@bi...
> web: http://taxonomy.zoology.gla.ac.uk/rod/rod.html
> iChat: aim://rodpage1962
> reprints: http://taxonomy.zoology.gla.ac.uk/rod/pubs.html
> 
> Subscribe to Systematic Biology through the Society of Systematic
> Biologists Website: http://systematicbiology.org
> Search for taxon names: http://darwin.zoology.gla.ac.uk/~rpage/portal/
> Find out what we know about a species: http://ispecies.org
> Rod's rants on phyloinformatics: http://iphylo.blogspot.com
> Rod's rants on ants: http://semant.blogspot.com
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> Connotea-code-devel mailing list
> Con...@li...
> https://lists.sourceforge.net/lists/listinfo/connotea-code-devel
> 





-- 
Ian Mulvany
i.m...@na...
+44(0)2078434756
www.nature.com
www.connotea.org
--



********************************************************************************   
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan 
Publishers Limited or its agents by means of e-mail communication. Macmillan 
Publishers Limited Registered in England and Wales with registered number 785998 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************