connotea-code-devel

[Connotea-code-devel] feedback

[Kieren D. <ki...@di...>]

Hi,

I've been wanting to write this email for a while, and you can  
consider me a strong supporter of what Connotea/NPG  is trying to  
do.  However there are some major problems that I think need to be  
addressed if connotea is to become a self sustaining open source  
project with a sustainable developer community.

A bit about me:  I'm a social researcher and perl hacker.  Most of  
the perl I write is for research data management, but I do  
bibliographic stuff, web robots and occasional web applications  
programming too.  I've been a significant contributor to the Catalyst  
web application framework (mainly with the documentation and example  
code).  I'm computer literate, and I've used mod_perl enough to have  
a good idea of what irritates me about it.  I think that Catalyst is  
the natural successor for 95% of the mod_perl or Apache::Registry  
scripts out there.

Problems I've experienced with connotea as an open source project.   
Suggested solutions in [square brackets].  Roughly in order of  
importance.

1.  No test suite.  Lack of unit/behaviour tests along with the  
impoverished debugging environment with mod_perl make for painful  
development. [ SOLUTION:  Port to catalyst with built in testing  
utilities and trivial support for perl -d ]

2.  Class::DBI + memcached.  These seem to be very tightly coupled.   
Class::DBI is flawed software, and while it has proved very useful,  
it clearly has severe limitations.  One example is the difficulty of  
inspecting the sql that's it's generating.  It's main problem is a  
general lack of transparency and implicitly generated code.  CDBI   
"died" as an ongoing open source project about a year ago.  The  
original author of CDBI's last project resulted in him fixing  
DBIx::Class' CDBI Compatibility layer  so he could port his client's  
code from CDBI  to DBIx::Class.

[SOLUTION:  implied by the last comment DBIx::Class (DBIC)  is the  
natural successor to CDBI, originally developed by the author of  
CDBI::Sweet.  DBIC + memcached has production users.  Discussion on  
IRC  (#dbix-class on irc.perl.org) indicates to me that swappable  
caching engines (including a null cache for debugging) ) should be  
trivial and transparent with a DBIC based data model.  The killer  
dbic feature for me is that running your dbic script like so:   
"DBIC_TRACE=1 perl run_my_script.pl" gives you the exact sql being  
passed to the dbic classes.  This outputs to standard error, thus  
making debugging hugely easier.  As noted, there's a CDBICompat layer  
to ease transition from CDBI to DBIC.  CDBICompat has a more  
extensive test suite than CDBI for good measure.  Finally DBIC tends  
to result in much more efficent SQL than CDBI and swapping out  
database engines (eg mysql to pg to sqlite) is much easier - I've  
seen this done trivially from pg to sqlite for a database  
representing a directed acyclic graph. ]

3.  Template toolkit.  The templates in connotea seem to be populated  
with weird coderefs that make debugging/interrogation even more  
difficult.  TT is one of the best templating solutions out there, but  
connetea seems to misuse it. [SOLUTION:  Port to explicit templates  
resident in files, using PROCESS, INCLUDE and MACRO blocks where  
appropriate.  This should be fairly simple to do concurrently with  
the rest of the catalyst/dbic port. ]

4.  Connotea doesn't scale down well.  I've used it on an ibook g4  
for testing (performance verging on reasonable), and a pentium 3  
linux machine with 128mb of RAM where performance was unacceptable.  
[SOLUTON:  port to catalyst where fast_cgi, mod_perl and other more  
exotic engines are all viable solutions.  Removing the mod_perl  
depencency opens up shared hosting possibilities]

There are some excellent things about connotea (auto-import and the  
database schema being two big examples), but the above are show  
stoppers which are going to cause huge problems for the  
sustainability of the project.  The usability for end users is great,  
but as far as programmer-usability goes connotea needs major  
improvements otherwise it doesn't have a future as a viable open  
source project.  Personally I'd like to hack in storage of pdf/other  
fulltext into the database, but I can't do this in it's current  
state.  I can also see why NPG won't do this themselves.

My vision is for distributed collaborative bibliographies, which is  
why I'd like so much for the project to scale down to a level where  
it could be used by 2-10 researchers on shared hosting.  I think  
optional  re-import back to a master connotea  would be fairly easy  
to implement after the scaling down problem was addressed.

Part of the problem is that connotea came about at a time of great  
flux during the web app programming space, and the technology to  
program these things has improved massively during this time.   
Unfortunately it leaves connotea with more technical debt.

I'm happy to clarify anything I've written here on request.  If the  
response is going to be "isn't going to happen, sorry about that",  
I'm prepared to accept that too.  As a first step I'd recommend  
popping on to #catalyst at irc.perl.org and asking about porting  
mod_perl apps to Catalyst.

--
Kieren Diment
Centre for Leadership and Knowledge Management
School of Management, Marketing

Re: [Connotea-code-devel] feedback

[Martin F. <ma...@ne...>]

Hi Kieren,

Thanks for your email. I'm responsible for most of the codebase so I 
will step into the firing line first. ;-)

Kieren Diment wrote:
> I've been wanting to write this email for a while, and you can  
> consider me a strong supporter of what Connotea/NPG  is trying to  
> do.  However there are some major problems that I think need to be  
> addressed if connotea is to become a self sustaining open source  
> project with a sustainable developer community.

Again thanks. I'm aware of most of what you raise, and in fact have 
discussed them with folks, but it doesn't hurt to acknowledge on the 
devel list.

Actually this works out nicely since we are about to release a new 
version of Connotea Code literally any day now, and we are committed to 
posting a public darcs repository and inviting more public support. I'm 
actually thinking you have seen the new code by some of your comments.

> A bit about me:  I'm a social researcher and perl hacker.  Most of  
> the perl I write is for research data management, but I do  
> bibliographic stuff, web robots and occasional web applications  
> programming too.  I've been a significant contributor to the Catalyst  
> web application framework (mainly with the documentation and example  
> code).  I'm computer literate, and I've used mod_perl enough to have  
> a good idea of what irritates me about it.  I think that Catalyst is  
> the natural successor for 95% of the mod_perl or Apache::Registry  
> scripts out there.

I've looked at Catalyst but I've not used it.

> 1.  No test suite.  Lack of unit/behaviour tests along with the  
> impoverished debugging environment with mod_perl make for painful  
> development. [ SOLUTION:  Port to catalyst with built in testing  
> utilities and trivial support for perl -d ]

The new version of Connotea Code sports a fledgling test suite. It 
doesn't shoot for code coverage, just to test the basic functions. It's 
an area I'd like to strengthen.

I concur on the mod_perl debugging situation.

> 2.  Class::DBI + memcached.  These seem to be very tightly coupled.   
> Class::DBI is flawed software, and while it has proved very useful,  
> it clearly has severe limitations.  One example is the difficulty of  
> inspecting the sql that's it's generating.  It's main problem is a  
> general lack of transparency and implicitly generated code.  CDBI   
> "died" as an ongoing open source project about a year ago.  The  
> original author of CDBI's last project resulted in him fixing  
> DBIx::Class' CDBI Compatibility layer  so he could port his client's  
> code from CDBI  to DBIx::Class.

Accurate observation. At the time it seemed like the best way to provide 
abstractions that made the model possible.

I'd add a point that you didn't make, which is that Bibliotech::Query is 
  very complicated, not very clear, and the queries it generates are 
huge, but to some extent it's by necessity.

For the type of thing like Connotea, Class::DBI is way too slow left to 
its own devices - a page was taking 800 queries to pull up all the 
pieces of data, and we solved that with a scary-looking but 
fast-executing query. Another problem might be that it was designed to 
be a very flexible query engine, and as it turns out that flexibility is 
obscuring some of the functionality, i.e. tightening the restraints 
might make prettier source code.

What we got from Class::DBI was a convenient abstraction that database 
rows would be objects that could relate to each other. Beyond that we've 
extended the heck out of it, and probably not canonically, although in 
our defense Class::DBI is only designed to have a canonical usage for 
the theoretical database model, not a whole application.

The new version of Connotea Code has a Bibliotech::Query with even more 
speed optimizations but again it's another layer of complexity.

> [SOLUTION:  implied by the last comment DBIx::Class (DBIC)  is the  
> natural successor to CDBI, originally developed by the author of  
> CDBI::Sweet.  DBIC + memcached has production users.  Discussion on  
> IRC  (#dbix-class on irc.perl.org) indicates to me that swappable  
> caching engines (including a null cache for debugging) ) should be  
> trivial and transparent with a DBIC based data model.  The killer  
> dbic feature for me is that running your dbic script like so:   
> "DBIC_TRACE=1 perl run_my_script.pl" gives you the exact sql being  
> passed to the dbic classes.  This outputs to standard error, thus  
> making debugging hugely easier.  As noted, there's a CDBICompat layer  
> to ease transition from CDBI to DBIC.  CDBICompat has a more  
> extensive test suite than CDBI for good measure.  Finally DBIC tends  
> to result in much more efficent SQL than CDBI and swapping out  
> database engines (eg mysql to pg to sqlite) is much easier - I've  
> seen this done trivially from pg to sqlite for a database  
> representing a directed acyclic graph. ]

I've got SQL tracing - everything except do() calls and without 
question-mark translation - if you call 
Bibliotech::DBI->activate_warn_sql(). In mod_perl this sends up in 
Apache's error_log.

Again I've hard of DBIx::Class but was not aware that it offered any 
major benefits.

> 3.  Template toolkit.  The templates in connotea seem to be populated  
> with weird coderefs that make debugging/interrogation even more  
> difficult.  TT is one of the best templating solutions out there, but  
> connetea seems to misuse it. [SOLUTION:  Port to explicit templates  
> resident in files, using PROCESS, INCLUDE and MACRO blocks where  
> appropriate.  This should be fairly simple to do concurrently with  
> the rest of the catalyst/dbic port. ]

Anything that looks like misuse of TT is probably a result of yielding 
functionality from our original Component system to TT, where the rest 
of the Component system is still there, and which was more capable than 
what we perceived TT could do, and we needed to preserve functionality.

It may be that TT could do more than we realized, or could not then but 
can now, and in that case it would be, as you point out, better to use 
the native TT functionality rather than homegrown functionality. I agree 
on principle.

As an example, our Component system was designed to put all the logic 
for a component in one place, and avoid expensive recalculation. To that 
end, a component's result is not just a snippet of HTML like an INCLUDE, 
but an object that contains HTML parts that can be put in different 
places on the calling web page, as well as the Javascript for the <head> 
and Javascript for <body onload=""> that facilitate the component, so if 
you want to, say, position the cursor in the first form field, you can 
have the component return the HTML form and that Javascript together.

The new Connotea Code release has a section in the README to cover the 
TT function calls which should explain things better.

> 4.  Connotea doesn't scale down well.  I've used it on an ibook g4  
> for testing (performance verging on reasonable), and a pentium 3  
> linux machine with 128mb of RAM where performance was unacceptable.  
> [SOLUTON:  port to catalyst where fast_cgi, mod_perl and other more  
> exotic engines are all viable solutions.  Removing the mod_perl  
> depencency opens up shared hosting possibilities]

We used to have a bibliotech.cgi that could do exactly that (using 
Apache::Emulator) but we dropped it because NPG had no need for it and 
it was after all just a hack. I'd be happy to share it with you if you 
are interested, but the added memcached requirement is the main obstacle 
I think.

> There are some excellent things about connotea (auto-import and the  
> database schema being two big examples), but the above are show  
> stoppers which are going to cause huge problems for the  
> sustainability of the project.  The usability for end users is great,  
> but as far as programmer-usability goes connotea needs major  
> improvements otherwise it doesn't have a future as a viable open  
> source project.  Personally I'd like to hack in storage of pdf/other  
> fulltext into the database, but I can't do this in it's current  
> state.  I can also see why NPG won't do this themselves.

Thanks for the compliments!

At least you're not complaining it's not Ruby on Rails. ;-)

That's a joke, but I acknowledge that CPAN marches forwards even after 
we have selected our libraries and yes platforms make a huge difference. 
We selected solutions to match the engineering and then we tried to move 
to some newer approaches as dictated by practical concerns. Even moving 
to TT was only a direct response to needing non-programmers to do 
editing and design work.

> My vision is for distributed collaborative bibliographies, which is  
> why I'd like so much for the project to scale down to a level where  
> it could be used by 2-10 researchers on shared hosting.  I think  
> optional  re-import back to a master connotea  would be fairly easy  
> to implement after the scaling down problem was addressed.

Your work sounds interesting.

> Part of the problem is that connotea came about at a time of great  
> flux during the web app programming space, and the technology to  
> program these things has improved massively during this time.   

You are saying something that sounds to me like:

- refactoring for better library support would make easier development

I'd also add:

- like all things, more man-hours on the codebase for general 
refactoring or more test suite scripts would make easier development, 
keeping the same libraries even

- refactoring to remove some early assumptions would make a cleaner 
codebase as well

...and I'd argue that those two points are just as important.

> Unfortunately it leaves connotea with more technical debt.

Well, I think you may be being a bit hard on us here. ;-)

This is a classic challenge facing companies with working code; whether 
or not to spend time refactoring things.

When allocating sparse resources carefully, while being customer driven, 
a lot of the push is inevitably on keeping it running and adding 
features rather than revising the abstractions for something that 
already works.

Further, things that may make programming easier but don't actually 
lower the complexity level of the codebase are not as appealing in 
general as refactoring projects; e.g. switching from Class::DBI to 
DBIx::Class is not going to make it easier to understand, (maybe) just 
easier to work with. The concepts don't change much, you still have an 
abstraction layer that provides SQL support. You and I can appreciate 
that, but companies generally are interested in refactoring code when it 
can introduce junior programmers, not just make it easier for the senior 
programmers.

Having said that, your suggestions that we may be doing something in a 
non-standard manner, if they can be done in a standard manner, I do 
consider bugs to fix, as we should do things the same way as everyone 
else where possible.

If Class::DBI -> DBIx::Class is thought of as that type of problem, then 
I can appreciate the desire to switch over. But I'm speaking here more 
of things like using homegrown approaches over TT directives, as 
discussed above.

> I'm happy to clarify anything I've written here on request.  If the  
> response is going to be "isn't going to happen, sorry about that",  
> I'm prepared to accept that too.  As a first step I'd recommend  
> popping on to #catalyst at irc.perl.org and asking about porting  
> mod_perl apps to Catalyst.

I'm not the final word, and even if I was, what you've presented 
deserves study and thought before issuing the final word. ;-)

I suggest we keep the conversation going.

Regards,
Martin Flack

Re: [Connotea-code-devel] feedback

[Kieren D. <ki...@di...>]

Thanks for the quick feedback to my feedback :).  This appears to be  
quite complicated, hence the top posting (sorry ;-) )

I'd love to hack on connotea but in it's present state I can't mainly  
because of the mod_perl dependency and the tight coupling of the CDBI  
code with the application logic. I'd like to have a look at  
connotea.cgi - if it's reasonably simple I might even be able to  
demonstrate a partial catalyst port (no guarantees).

Fundamentally the connotea  codebase lacks the MVC pattern.  Moving  
to a more MVC architecture with test coverage is going to make life  
easier for everyone, senior and junior developers alike, as well as  
casual hackers like me.   So as a second iteration I'm going to deal  
with this as an MVC problem (CMV actually)

1.  Controller

  Porting to Catalyst provides the framework for this, with sane url/ 
action dispatching (I couldn't work out how this works looking at the  
connotea code base  with my limited available time.  This is another  
problem).  Using the catalyst dispatcher makes it very quick for new  
developers to get their head around the flow of logic within the  
application (the controller part).  The main task here is refactoring  
the mod_perl handler dispatching logic into something catalyst  
compatible.  I've seen people doing this a few times, and it should  
be pretty quick and straightforward for someone familliar with the  
existing mod_perl code base.  An experienced mod_perl hacker (which  
I'm not) from the catalyst project was going to have a look at the  
connotea code base for me to tell me if my assumptions above are  
correct, but because there's no code viewable in a web browser, this  
proved too much friction ... for now.  (Here illustrates a major  
benifit of open source software communities - a lot of my comments  
don't come from my expertise, but from knowing how to ask the right  
questions on the appropriate irc channel).

A major benefit of catalyst  is that perl -d script/ 
connotea_server.pl (with appropriate $DB::single=1 statements in your  
code) provides excellent debugging support.  I suspect that porting  
to catalyst would pay itself back in reduced development and training  
time very quickly.

2.  Model

DBIC is much more flexible and much faster than CDBI.  A port would  
again reduce technical overhead.  However if you can get the CDBI  
model working independently of the mod_perl app, it becomes usable  
within catalyst, and you can defer the decision to port while  
improving the structure of the codebase.  However, the tight coupling  
of the cdbi models with memcached is another impediment to casual  
hacking on connotea.  I suspect that porting to DBIC would make life  
a lot easier for junior and senior developers alike, and be the  
easiest way to get a web-app-independent model available as well.   
Thanks for reminding me about the sql trace stuff you added.  It was  
a while ago, but not being able to see the do() stuff was  a show  
stopper for me I'm afraid.

3.  View

Based on your feedback, if using  catalyst, you'd probably want to  
make a souped up Catalyst::View::ConnoteaTT based on  
Catalyst::View::TT.  Maybe also  make a  Catalyst::View::RDF as  
well.  Improving the architecture to something more MVC is going to  
make all these little components much more modular and easier for new  
(junior and senior) developers alike to understand.

A core problem here is that with the current architecture/libraries  
of connotea you don't have very good ability to leverage the perl  
community to help with development.  You should also upload the  
connotea perl api libraries to CPAN so that connotea becomes more  
visible to the perl community at large, and make a post to  
use.perl.org once the new version is out with it's vc repository.

I didn't suggest Rails because Catalyst is far superior both in  
library support and that it's much less opinionated, and thus has the  
flexibility that connotea requires. :-D

Kieren

On 9 Aug 2007, at 05:00, Martin Flack wrote:

> Hi Kieren,
>
> Thanks for your email. I'm responsible for most of the codebase so  
> I will step into the firing line first. ;-)
>
> Kieren Diment wrote:
>> I've been wanting to write this email for a while, and you can   
>> consider me a strong supporter of what Connotea/NPG  is trying to   
>> do.  However there are some major problems that I think need to  
>> be  addressed if connotea is to become a self sustaining open  
>> source  project with a sustainable developer community.
>
> Again thanks. I'm aware of most of what you raise, and in fact have  
> discussed them with folks, but it doesn't hurt to acknowledge on  
> the devel list.
>
> Actually this works out nicely since we are about to release a new  
> version of Connotea Code literally any day now, and we are  
> committed to posting a public darcs repository and inviting more  
> public support. I'm actually thinking you have seen the new code by  
> some of your comments.
>
>> A bit about me:  I'm a social researcher and perl hacker.  Most  
>> of  the perl I write is for research data management, but I do   
>> bibliographic stuff, web robots and occasional web applications   
>> programming too.  I've been a significant contributor to the  
>> Catalyst  web application framework (mainly with the documentation  
>> and example  code).  I'm computer literate, and I've used mod_perl  
>> enough to have  a good idea of what irritates me about it.  I  
>> think that Catalyst is  the natural successor for 95% of the  
>> mod_perl or Apache::Registry  scripts out there.
>
> I've looked at Catalyst but I've not used it.
>
>> 1.  No test suite.  Lack of unit/behaviour tests along with the   
>> impoverished debugging environment with mod_perl make for painful   
>> development. [ SOLUTION:  Port to catalyst with built in testing   
>> utilities and trivial support for perl -d ]
>
> The new version of Connotea Code sports a fledgling test suite. It  
> doesn't shoot for code coverage, just to test the basic functions.  
> It's an area I'd like to strengthen.
>
> I concur on the mod_perl debugging situation.
>
>> 2.  Class::DBI + memcached.  These seem to be very tightly  
>> coupled.   Class::DBI is flawed software, and while it has proved  
>> very useful,  it clearly has severe limitations.  One example is  
>> the difficulty of  inspecting the sql that's it's generating.   
>> It's main problem is a  general lack of transparency and  
>> implicitly generated code.  CDBI   "died" as an ongoing open  
>> source project about a year ago.  The  original author of CDBI's  
>> last project resulted in him fixing  DBIx::Class' CDBI  
>> Compatibility layer  so he could port his client's  code from  
>> CDBI  to DBIx::Class.
>
> Accurate observation. At the time it seemed like the best way to  
> provide abstractions that made the model possible.
>
> I'd add a point that you didn't make, which is that  
> Bibliotech::Query is  very complicated, not very clear, and the  
> queries it generates are huge, but to some extent it's by necessity.
>
> For the type of thing like Connotea, Class::DBI is way too slow  
> left to its own devices - a page was taking 800 queries to pull up  
> all the pieces of data, and we solved that with a scary-looking but  
> fast-executing query. Another problem might be that it was designed  
> to be a very flexible query engine, and as it turns out that  
> flexibility is obscuring some of the functionality, i.e. tightening  
> the restraints might make prettier source code.
>
> What we got from Class::DBI was a convenient abstraction that  
> database rows would be objects that could relate to each other.  
> Beyond that we've extended the heck out of it, and probably not  
> canonically, although in our defense Class::DBI is only designed to  
> have a canonical usage for the theoretical database model, not a  
> whole application.
>
> The new version of Connotea Code has a Bibliotech::Query with even  
> more speed optimizations but again it's another layer of complexity.
>
>> [SOLUTION:  implied by the last comment DBIx::Class (DBIC)  is  
>> the  natural successor to CDBI, originally developed by the author  
>> of  CDBI::Sweet.  DBIC + memcached has production users.   
>> Discussion on  IRC  (#dbix-class on irc.perl.org) indicates to me  
>> that swappable  caching engines (including a null cache for  
>> debugging) ) should be  trivial and transparent with a DBIC based  
>> data model.  The killer  dbic feature for me is that running your  
>> dbic script like so:   "DBIC_TRACE=1 perl run_my_script.pl" gives  
>> you the exact sql being  passed to the dbic classes.  This outputs  
>> to standard error, thus  making debugging hugely easier.  As  
>> noted, there's a CDBICompat layer  to ease transition from CDBI to  
>> DBIC.  CDBICompat has a more  extensive test suite than CDBI for  
>> good measure.  Finally DBIC tends  to result in much more efficent  
>> SQL than CDBI and swapping out  database engines (eg mysql to pg  
>> to sqlite) is much easier - I've  seen this done trivially from pg  
>> to sqlite for a database  representing a directed acyclic graph. ]
>
> I've got SQL tracing - everything except do() calls and without  
> question-mark translation - if you call Bibliotech::DBI- 
> >activate_warn_sql(). In mod_perl this sends up in Apache's error_log.
>
> Again I've hard of DBIx::Class but was not aware that it offered  
> any major benefits.
>
>> 3.  Template toolkit.  The templates in connotea seem to be  
>> populated  with weird coderefs that make debugging/interrogation  
>> even more  difficult.  TT is one of the best templating solutions  
>> out there, but  connetea seems to misuse it. [SOLUTION:  Port to  
>> explicit templates  resident in files, using PROCESS, INCLUDE and  
>> MACRO blocks where  appropriate.  This should be fairly simple to  
>> do concurrently with  the rest of the catalyst/dbic port. ]
>
> Anything that looks like misuse of TT is probably a result of  
> yielding functionality from our original Component system to TT,  
> where the rest of the Component system is still there, and which  
> was more capable than what we perceived TT could do, and we needed  
> to preserve functionality.
>
> It may be that TT could do more than we realized, or could not then  
> but can now, and in that case it would be, as you point out, better  
> to use the native TT functionality rather than homegrown  
> functionality. I agree on principle.
>
> As an example, our Component system was designed to put all the  
> logic for a component in one place, and avoid expensive  
> recalculation. To that end, a component's result is not just a  
> snippet of HTML like an INCLUDE, but an object that contains HTML  
> parts that can be put in different places on the calling web page,  
> as well as the Javascript for the <head> and Javascript for <body  
> onload=""> that facilitate the component, so if you want to, say,  
> position the cursor in the first form field, you can have the  
> component return the HTML form and that Javascript together.
>
> The new Connotea Code release has a section in the README to cover  
> the TT function calls which should explain things better.
>
>> 4.  Connotea doesn't scale down well.  I've used it on an ibook  
>> g4  for testing (performance verging on reasonable), and a pentium  
>> 3  linux machine with 128mb of RAM where performance was  
>> unacceptable.  [SOLUTON:  port to catalyst where fast_cgi,  
>> mod_perl and other more  exotic engines are all viable solutions.   
>> Removing the mod_perl  depencency opens up shared hosting  
>> possibilities]
>
> We used to have a bibliotech.cgi that could do exactly that (using  
> Apache::Emulator) but we dropped it because NPG had no need for it  
> and it was after all just a hack. I'd be happy to share it with you  
> if you are interested, but the added memcached requirement is the  
> main obstacle I think.
>
>> There are some excellent things about connotea (auto-import and  
>> the  database schema being two big examples), but the above are  
>> show  stoppers which are going to cause huge problems for the   
>> sustainability of the project.  The usability for end users is  
>> great,  but as far as programmer-usability goes connotea needs  
>> major  improvements otherwise it doesn't have a future as a viable  
>> open  source project.  Personally I'd like to hack in storage of  
>> pdf/other  fulltext into the database, but I can't do this in it's  
>> current  state.  I can also see why NPG won't do this themselves.
>
> Thanks for the compliments!
>
> At least you're not complaining it's not Ruby on Rails. ;-)
>
> That's a joke, but I acknowledge that CPAN marches forwards even  
> after we have selected our libraries and yes platforms make a huge  
> difference. We selected solutions to match the engineering and then  
> we tried to move to some newer approaches as dictated by practical  
> concerns. Even moving to TT was only a direct response to needing  
> non-programmers to do editing and design work.
>
>> My vision is for distributed collaborative bibliographies, which  
>> is  why I'd like so much for the project to scale down to a level  
>> where  it could be used by 2-10 researchers on shared hosting.  I  
>> think  optional  re-import back to a master connotea  would be  
>> fairly easy  to implement after the scaling down problem was  
>> addressed.
>
> Your work sounds interesting.
>
>> Part of the problem is that connotea came about at a time of  
>> great  flux during the web app programming space, and the  
>> technology to  program these things has improved massively during  
>> this time.
>
> You are saying something that sounds to me like:
>
> - refactoring for better library support would make easier development
>
> I'd also add:
>
> - like all things, more man-hours on the codebase for general  
> refactoring or more test suite scripts would make easier  
> development, keeping the same libraries even
>
> - refactoring to remove some early assumptions would make a cleaner  
> codebase as well
>
> ...and I'd argue that those two points are just as important.
>
>> Unfortunately it leaves connotea with more technical debt.
>
> Well, I think you may be being a bit hard on us here. ;-)
>
> This is a classic challenge facing companies with working code;  
> whether or not to spend time refactoring things.
>
> When allocating sparse resources carefully, while being customer  
> driven, a lot of the push is inevitably on keeping it running and  
> adding features rather than revising the abstractions for something  
> that already works.
>
> Further, things that may make programming easier but don't actually  
> lower the complexity level of the codebase are not as appealing in  
> general as refactoring projects; e.g. switching from Class::DBI to  
> DBIx::Class is not going to make it easier to understand, (maybe)  
> just easier to work with. The concepts don't change much, you still  
> have an abstraction layer that provides SQL support. You and I can  
> appreciate that, but companies generally are interested in  
> refactoring code when it can introduce junior programmers, not just  
> make it easier for the senior programmers.
>
> Having said that, your suggestions that we may be doing something  
> in a non-standard manner, if they can be done in a standard manner,  
> I do consider bugs to fix, as we should do things the same way as  
> everyone else where possible.
>
> If Class::DBI -> DBIx::Class is thought of as that type of problem,  
> then I can appreciate the desire to switch over. But I'm speaking  
> here more of things like using homegrown approaches over TT  
> directives, as discussed above.
>
>> I'm happy to clarify anything I've written here on request.  If  
>> the  response is going to be "isn't going to happen, sorry about  
>> that",  I'm prepared to accept that too.  As a first step I'd  
>> recommend  popping on to #catalyst at irc.perl.org and asking  
>> about porting  mod_perl apps to Catalyst.
>
> I'm not the final word, and even if I was, what you've presented  
> deserves study and thought before issuing the final word. ;-)
>
> I suggest we keep the conversation going.
>
> Regards,
> Martin Flack
>

Re: [Connotea-code-devel] feedback

[Martin F. <ma...@ne...>]

Hi Kieren,

A couple quick responses.

Kieren Diment wrote:
> 1.  Controller
> 
>   Porting to Catalyst provides the framework for this, with sane url/ 
> action dispatching (I couldn't work out how this works looking at the  
> connotea code base  with my limited available time.  This is another  
> problem).  Using the catalyst dispatcher makes it very quick for new  
> developers to get their head around the flow of logic within the  
> application (the controller part).  The main task here is refactoring  
> the mod_perl handler dispatching logic into something catalyst  
> compatible.  I've seen people doing this a few times, and it should  
> be pretty quick and straightforward for someone familliar with the  
> existing mod_perl code base.  An experienced mod_perl hacker (which  
> I'm not) from the catalyst project was going to have a look at the  
> connotea code base for me to tell me if my assumptions above are  
> correct, but because there's no code viewable in a web browser, this  
> proved too much friction ... for now.  (Here illustrates a major  
> benifit of open source software communities - a lot of my comments  
> don't come from my expertise, but from knowing how to ask the right  
> questions on the appropriate irc channel).

A brief sketch of the current dispatch:

Bibliotech::Apache runs the dispatch. Bibliotech::Parser understands the 
URI passed in, and returns a Bibliotech::Command object. An appropriate 
Bibliotech::Page object is created, which provides a last_updated() 
method and an html_content() method that return results (similarly, it 
is ris_content() or rss_content() etc. for other formats). The 
html_content() method generally uses TT and Bibliotech::Query and 
various Bibliotech::Component's and some help from Bibliotech::DBI layer 
and Bibliotech::Util. Bibliotech::DBI is the base class for all objects 
in the database, and yes they all have an html_content() as well. 
Actions such as adding and editing have root functions in Bibliotech.pm 
since they coordinate across many modules.

I think you'd get your CMV separation as good as any application can get 
with these steps:

- removing the *_content() methods and supporting items and making them 
subclasses of the Bibliotech::Page and Bibliotech::Component objects 
with a content() method. That just seemed complicated

- removing the *_content() methods and supporting items from 
Bibliotech::DBI objects and putting them into similarly separated 
subclasses or have another class that accepts objects and renders them 
or something like that.

It just seemed like that increased syntactical complexity and class 
complexity in order to create separation that we didn't think we needed 
when the codebase was smaller.

I'm actually interested in seeing a fairly big application written in 
Catalyst to see how you guys do it.

> 2.  Model
> 
> DBIC is much more flexible and much faster than CDBI.  A port would  
> again reduce technical overhead.  However if you can get the CDBI  
> model working independently of the mod_perl app, it becomes usable  
> within catalyst, and you can defer the decision to port while  
> improving the structure of the codebase.  However, the tight coupling  
> of the cdbi models with memcached is another impediment to casual  
> hacking on connotea.  I suspect that porting to DBIC would make life  
> a lot easier for junior and senior developers alike, and be the  
> easiest way to get a web-app-independent model available as well.   
> Thanks for reminding me about the sql trace stuff you added.  It was  
> a while ago, but not being able to see the do() stuff was  a show  
> stopper for me I'm afraid.

Unfortunately we are not even within the bounds of Class::DBI because 
Bibliotech::Query does a lot of query building itself, although I am 
interested in revamping it because it is the most difficult part of the 
codebase. So we'd probably have to do a general refactoring before we 
even switched.

> 3.  View
> 
> Based on your feedback, if using  catalyst, you'd probably want to  
> make a souped up Catalyst::View::ConnoteaTT based on  
> Catalyst::View::TT.  Maybe also  make a  Catalyst::View::RDF as  
> well.  Improving the architecture to something more MVC is going to  
> make all these little components much more modular and easier for new  
> (junior and senior) developers alike to understand.
> 
> A core problem here is that with the current architecture/libraries  
> of connotea you don't have very good ability to leverage the perl  
> community to help with development.  You should also upload the  
> connotea perl api libraries to CPAN so that connotea becomes more  
> visible to the perl community at large, and make a post to  
> use.perl.org once the new version is out with it's vc repository.
> 
> I didn't suggest Rails because Catalyst is far superior both in  
> library support and that it's much less opinionated, and thus has the  
> flexibility that connotea requires. :-D

All noted. Thanks.

Martin

[Connotea-code-devel] MVC.

[Kieren D. <ki...@di...>]

On 10 Aug 2007, at 00:33, Martin Flack wrote:

> A brief sketch of the current dispatch:
>
> Bibliotech::Apache runs the dispatch. Bibliotech::Parser  
> understands the URI passed in, and returns a Bibliotech::Command  
> object. An appropriate Bibliotech::Page object is created, which  
> provides a last_updated() method and an html_content() method that  
> return results (similarly, it is ris_content() or rss_content()  
> etc. for other formats). The html_content() method generally uses  
> TT and Bibliotech::Query and various Bibliotech::Component's and  
> some help from Bibliotech::DBI layer and Bibliotech::Util.  
> Bibliotech::DBI is the base class for all objects in the database,  
> and yes they all have an html_content() as well. Actions such as  
> adding and editing have root functions in Bibliotech.pm since they  
> coordinate across many modules.
>

Riight

I think that you could do a fairly straightforward refactor to  
catalyst by:

1.  Getting the database code working independently of the web  
application and calling it a well separated model.

2.  Taking your dispatcher code, getting it working independent of  
apache (basically factoring out $r into utility functions).  Next up,  
delegating the dispatching logic and content generation logic below  
and bending it into catalyst logic.  I really think this isn't going  
hard for someone familiar with the codebase, although it would be  
tricky for me.  What this buys you is a clearly understood dispatch  
process that's very easily understood by a large and active developer  
community.

> I think you'd get your CMV separation as good as any application  
> can get with these steps:
>
> - removing the *_content() methods and supporting items and making  
> them subclasses of the Bibliotech::Page and Bibliotech::Component  
> objects with a content() method. That just seemed complicated
>
> - removing the *_content() methods and supporting items from  
> Bibliotech::DBI objects and putting them into similarly separated  
> subclasses or have another class that accepts objects and renders  
> them or something like that.
>
> It just seemed like that increased syntactical complexity and class  
> complexity in order to create separation that we didn't think we  
> needed when the codebase was smaller.
>

mmm.  I really think that's called technical debt

> I'm actually interested in seeing a fairly big application written  
> in Catalyst to see how you guys do it.

right.  The really big ones tend to be closed source.  A prominent  
one ( Vox from 6apart):  http://tokyo2007.yapcasia.org/sessions/ 
2007/02/everything_vox.html

and a little experiment:

/tmp$ du -h -d 1 connotea-code-1.7.1
1.0M    connotea-code-1.7.1/bibliotech
1.0M    connotea-code-1.7.1

/tmp$ cd ~/Deskop/dev/Catalyst/trunk/examples # (http:// 
dev.catalyst.perl.org/repos/Catalyst/trunk/examples/)

~/Desktop/dev/Catalyst/trunk/examples$ du -h -d1
# I've edited out the ones I think are of no use to you of I don't  
know anything about
136K    ./AdventREST # example REST app
616K    ./CatalystAdvent # lots of example driven documentation
152K    ./CatTube # youtube integration
216K    ./ChainedEg # innovative dispatch method example - may be  
relevant to connotea may not
132K    ./Cheer # very simple tutorial app
144K    ./GeoCat # geotagging
488K    ./InstantMessenger # neat IM program - old so contains some  
deprecated approaches
156K    ./JQChat # minimal DBIC/Query chat program
152K    ./MiniMojoDBIC # minimal DBIC/Prototype wiki
172K    ./OpenID # openID integration
412K    ./ServerDB # server management software
3.2M    ./SmokeServer # server for testing software?
40K    ./Streaming # simple streaming app
352K    ./Tutorial # reference implementaiton for the catalyst  
tutorial in Catalyst::Manual::Tutorial
9.2M    .

Here's my Website in a Box which I wrote while learning to provide a  
complete and useful calalyst app with good test coverage and good  
programmer documentation.  The dispatch logic here is very simple.   
http://code.google.com/p/websiteinabox (codebase = 1.3M).  It has  
some flaws (the controller is too thick in places and the model too  
thin mainly).

Now onto the Model:

> Unfortunately we are not even within the bounds of Class::DBI  
> because Bibliotech::Query does a lot of query building itself,  
> although I am interested in revamping it because it is the most  
> difficult part of the codebase. So we'd probably have to do a  
> general refactoring before we even switched.
>

Right.   DBIC was made because projects outgrew CDBI very quickly in  
exactly this manner. In this case I'd really recommend doing a  
concurrent refactor to DBIx::Class to get a model that works  
independently of the app.  Getting a starting schema with good  
relationships coverage really is as simple as installing DBIx::Class  
and doing something like:

perl -MDBIx::Class::Schema::Loader=make_schema_at,dump_to_dir:/path/ 
to/lib -e  \
'make_schema_at("MySchemaName", {relationships => 1},  
["dbi:mysql:dbname=db", "user", "pass"])'

You've got a lot of sql you can then port back in by using the scalar  
ref feature in SQL::Abstract, and from there you can wind back to  
something a bit easier to maintain using core dbic functionality  
wherever possible.

chainable resultsets are a feature everybody finds extremely useful:

my $schema  = DB->connect();
  my $resultset = $schema->resultset('Foo')->search(\% 
scary_search_condition, \%search_attributes);
my $narrowed_resultset = $resultset->search(\%some_more_search_logic);
while (my $foo = $narrowed_resultset->next) { # you only hit the  
database at this point in the chaining of the resultset }

things like using a cache engine ie memcached should be transparent  
and trivial to remove for users who don't need that feature with DBIC  
(c.f.  tight couplling and  real maintenance headache for the  
equivalent CDBI code).

DBIx:Class ought to see the following schema from the connotea db  by  
default (ie relationships pre-specified for you in the generated  
code) - http://www.uow.edu.au/~kdiment/schema.png (generated by  
sqlfairy which DBIC uses for some purposes). I can't guarantee this,  
but if not it will be close.

As you can see I can continue this conversation for ages :-)

Kieren

Re: [Connotea-code-devel] MVC.

[Martin F. <ma...@ne...>]

Kieren, thanks for these comments.

The bulk of our work is on citation/import/export, speed optimization, 
user features, etc. so I am not averse at all to having some of the 
structural points reexamined, and you're probably correct that we 
haven't thought about them questioningly in a long time. If this is an 
issue for newcomers I'd like to be aware of it.

Kieren Diment wrote:
> 1.  Getting the database code working independently of the web  
> application and calling it a well separated model.

Your WIAB project looks file-based; I'd be interested in a similarly 
small SQL project.

> 2.  Taking your dispatcher code, getting it working independent of  
> apache (basically factoring out $r into utility functions).  Next up,  
> delegating the dispatching logic and content generation logic below  
> and bending it into catalyst logic.  I really think this isn't going  
> hard for someone familiar with the codebase, although it would be  
> tricky for me.  What this buys you is a clearly understood dispatch  
> process that's very easily understood by a large and active developer  
> community.

$r is basically only handled in Apache.pm. You can write a test script 
that creates a Bibliotech::Command or uses Bibliotech::Parser to create 
one. Bibliotech::Fake was created for this purpose.

Having said that, you are correct it would be good to add more 
abstraction to make this easier, and make command-line uses easier.

I suppose at the moment I am perhaps so familiar with the codebase I can 
write a test script "too quickly."

>> I think you'd get your CMV separation as good as any application  
>> can get with these steps:
>>
>> - removing the *_content() methods and supporting items and making  
>> them subclasses of the Bibliotech::Page and Bibliotech::Component  
>> objects with a content() method. That just seemed complicated
>>
>> - removing the *_content() methods and supporting items from  
>> Bibliotech::DBI objects and putting them into similarly separated  
>> subclasses or have another class that accepts objects and renders  
>> them or something like that.
>>
>> It just seemed like that increased syntactical complexity and class  
>> complexity in order to create separation that we didn't think we  
>> needed when the codebase was smaller.
>>
> 
> mmm.  I really think that's called technical debt

Perhaps an explicit CMV remodel like this will turn out to be the first 
suggestion offered by you that we would try to implement then.

>> I'm actually interested in seeing a fairly big application written  
>> in Catalyst to see how you guys do it.
> 
> right.  The really big ones tend to be closed source.  A prominent  
> one ( Vox from 6apart):  http://tokyo2007.yapcasia.org/sessions/ 
> 2007/02/everything_vox.html

I certainly will take a look when I have time.

> and a little experiment:
> 
> /tmp$ du -h -d 1 connotea-code-1.7.1
> 1.0M    connotea-code-1.7.1/bibliotech
> 1.0M    connotea-code-1.7.1

I don't understand what your actual point here is though.

> Here's my Website in a Box which I wrote while learning to provide a  
> complete and useful calalyst app with good test coverage and good  
> programmer documentation.  The dispatch logic here is very simple.   
> http://code.google.com/p/websiteinabox (codebase = 1.3M).  It has  
> some flaws (the controller is too thick in places and the model too  
> thin mainly).

Interesting.

> Now onto the Model:
> 
>> Unfortunately we are not even within the bounds of Class::DBI  
>> because Bibliotech::Query does a lot of query building itself,  
>> although I am interested in revamping it because it is the most  
>> difficult part of the codebase. So we'd probably have to do a  
>> general refactoring before we even switched.
>>
> 
> Right.   DBIC was made because projects outgrew CDBI very quickly in  
> exactly this manner. In this case I'd really recommend doing a  
> concurrent refactor to DBIx::Class to get a model that works  
> independently of the app.  Getting a starting schema with good  
> relationships coverage really is as simple as installing DBIx::Class  
> and doing something like:

I'm not convinced your examples are as aggressive as we are in the SQL 
area. Just for giggles, I'm going to post a query that collects the data 
for /user/martin while logged in as martin. This will give you an idea 
of the consolidation that was necessary to collect the data for one page 
of Connotea in a reasonable number of seconds. A similar set of queries 
to collect all the requisite data was over 800 queries (really) when 
leaving Class::DBI to get the data itself so this is what it looks like 
now after packing it into one monster query that runs quickly. I'll post 
in a separate email.

I'm not saying you're wrong, just take a look if you like.

> You've got a lot of sql you can then port back in by using the scalar  
> ref feature in SQL::Abstract, and from there you can wind back to  
> something a bit easier to maintain using core dbic functionality  
> wherever possible.

BTW, we did use SQL::Abstract, so we do know about it.

> chainable resultsets are a feature everybody finds extremely useful:
> 
> my $schema  = DB->connect();
>   my $resultset = $schema->resultset('Foo')->search(\% 
> scary_search_condition, \%search_attributes);
> my $narrowed_resultset = $resultset->search(\%some_more_search_logic);
> while (my $foo = $narrowed_resultset->next) { # you only hit the  
> database at this point in the chaining of the resultset }

Ok. I'll have to study that class.

> things like using a cache engine ie memcached should be transparent  
> and trivial to remove for users who don't need that feature with DBIC  
> (c.f.  tight couplling and  real maintenance headache for the  
> equivalent CDBI code).

It is abstracted in Bibliotech::Cache so maybe just a 
Bibliotech::Cache::None or Bibliotech::Cache::Memory gets added to the 
project? A little bit of the caching is useful in the same request.

Do you happen to know of any public Catalyst projects that are required 
to use server-wide caching to handle the traffic load? So we could see 
their programming model for the cache interface? That would be interesting.

I'd note we cache in M and V stages in Connotea. Query results as 
constituted objects as well as component HTML and whole pages (HTML or 
otherwise) are cached, for visitors, and for individuals (since privacy 
affects everything we show in Connotea).

> DBIx:Class ought to see the following schema from the connotea db  by  
> default (ie relationships pre-specified for you in the generated  
> code) - http://www.uow.edu.au/~kdiment/schema.png (generated by  
> sqlfairy which DBIC uses for some purposes). I can't guarantee this,  
> but if not it will be close.

Sure.

Cheers,
Martin

Re: [Connotea-code-devel] MVC.

[Kieren D. <ki...@di...>]

On 11 Aug 2007, at 02:10, Martin Flack wrote:

> Kieren, thanks for these comments.
>
> The bulk of our work is on citation/import/export, speed  
> optimization, user features, etc. so I am not averse at all to  
> having some of the structural points reexamined, and you're  
> probably correct that we haven't thought about them questioningly  
> in a long time. If this is an issue for newcomers I'd like to be  
> aware of it.

I had a good long look at connotea about a year ago, but with my  
limited time, it was just too hard for me.  I would really like to  
use it again for a new project now, but I can't make full use of it  
for now, for the reasons I gave before.

I still really want to persuade you to refactor to DBIx::Class, and I  
think that this process may even be quicker than refactoring the cdbi  
code on its own, and certainly more developer/community  friendly.   
Bear in mind that many DBIx::Class developers are former cdbi  
developers and its design is informed by a knowledge of cdbi's  
limitations.  In connotea the model is the largest software  problem  
to be solved, and while you've solved it well in many ways, it's very  
hard for casual developers to pick it up and run with it.  I think  
DBIx::Class would provide much of a solution to this - the biggest  
feature in my book being DBIC_TRACE=1 myscript.pl producing all the  
sql used to STDERR

I've attached a tarball of some DBIC/Catalyst::Model code from the  
LinkMine (social bookmarking) project which I referred to in an  
earlier email.  It's not working code as such, and I'm not completely  
convinced that it follows best practice in all cases, but it shows  
the general idea.  I've also greped for all instances where the model  
is called in the controller code too.  I had to do some slightly  
exotic stuff to get a mysql database schema.  Basically I took the  
DBIx::Class schema declared in LinkMSchema (which was written for  
postgres originally) and ran the following script:

#!/usr/bin/perl
use warnings; use strict;
use LinkMSchema; # DBIx::Class schema
system('sqlite3 db'); # eugh, hack needs connect_info a blank sqlite  
db will do
my $schema = LinkMSchema->connect( "dbi:SQLite:db");
$schema->create_ddl_dir('MySQL', undef, './');

which provides me with the sql for a mysql schema.  I did lose a  
couple of triggers, but that's not important for illustrative purposes.

I found the horrible query you mentioned in Bibliotech::DB.pm .  You  
don't have to work around the limitations of DBIx::Class to implement  
this, it knows it's own limitations.  Here's the perldoc that sums up  
the situation with Connotea perfectly:  http://search.cpan.org/ 
~mstrout/DBIx-Class-0.08005/lib/DBIx/Class/Manual/ 
Cookbook.pod#Arbitrary_SQL_through_a_custom_ResultSource

Back in the day when I was using CDBI I didn't use it much because I  
found it so hard going, with uninformative errors, and documentation  
that I found difficult to follow.  It also has a habit of deep  
resucsion - you can demonstrate this with the connotea code base by  
adding Sub::WrapPackages into an appropriate place for debugging (I  
put it in Bibliotech::Apache).  Then DBIx::Class came along which has  
learned from the mistakes of CDBI.  Bear in mind my database usage is  
generally pretty limited, with little or no need for optimisation on  
my side so a lot of the DBIC code I write looks like this (in memory  
loading of the schema):

#!/usr/bin/perl -w
use DBIx::Class::Schema::Loader qw/make_schema_at/;
make_schema_at( 'MyDB', {relationships =>1, debug =>0, auto_update =>1},
                 [ 'dbi:mysql:database=mydb', 'user', 'pass'],);
my $schema= MyDB->connect();
my $rs = $schema->resultset('Table')->search({});
while (my $rec = $rs->next()) {
     # do stuff
}


>> 2.  Taking your dispatcher code, getting it working independent  
>> of  apache (basically factoring out $r into utility functions).   
>> Next up,  delegating the dispatching logic and content generation  
>> logic below  and bending it into catalyst logic.  I really think  
>> this isn't going  hard for someone familiar with the codebase,  
>> although it would be  tricky for me.  What this buys you is a  
>> clearly understood dispatch  process that's very easily understood  
>> by a large and active developer  community.
>
> $r is basically only handled in Apache.pm. You can write a test  
> script that creates a Bibliotech::Command or uses  
> Bibliotech::Parser to create one. Bibliotech::Fake was created for  
> this purpose.
>
> Having said that, you are correct it would be good to add more  
> abstraction to make this easier, and make command-line uses easier.
>

Indeed.  Also any example command line scripts would be much  
appreciated this would also help to begin to address the mod_perl  
dependency, which is a problem for widespread use of your code.  The  
mod_perl dependency is a show stopper for me - if I get a working  
standalone model with example code, I'm much more likely to hack at  
it (although CDBI still leaves me inclined to avoid unless there's  
lots of clear example code).


> things like using a cache engine ie memcached should be  
> transparent  and trivial to remove for users who don't need that  
> feature with DBIC  (c.f.  tight couplling and  real maintenance  
> headache for the  equivalent CDBI code).
>
> It is abstracted in Bibliotech::Cache so maybe just a  
> Bibliotech::Cache::None or Bibliotech::Cache::Memory gets added to  
> the project? A little bit of the caching is useful in the same  
> request.

If you're going to do that, make life easy on your developers and aim  
for total transparrency - so a null cache, then a slightly lighter  
cache if that proves necessary. Then memcached.

>
> Do you happen to know of any public Catalyst projects that are  
> required to use server-wide caching to handle the traffic load? So  
> we could see their programming model for the cache interface? That  
> would be interesting.
>

There are a few catalyst plugins on CPAN  that provide simple  
caching.  These can be modified for more complex use-cases of  
course.  These include component caching (see Catalyst::Plugin::Cache  
which has a bare bones catalyst app as part of its test suite), page  
cache (Catalyst::Plugin::PageCache - again with good app based test  
coverage).  Catalyst::Plugin::Cache::Memcached lacks this  
unfortunately. The core of caching with DBIx::Class (with the  
swappable backends) is DBIx::Class::Cursor::Cached.  TT also does  
some memory caching of its own.

Something else that catalyst buys you is the ready-rolled ability to  
distribute your app as a CPAN dist.  This makes installation pretty  
easy assuming you have your Makefile.PL set out properly:

$ perl Makefile.PL
$ make installdeps # or sudo make installdeps

Also as far as catalyst goes, there's a well developed set of (again  
multiple backend)  authentication/authorisation code.  However it's  
obvious that the data store for your auth/authz is in mysql.

(mail cc'd to your personal address in case the tarball gets stripped  
by sourceforge).

Kieren

Re: [Connotea-code-devel] feedback

[Kieren D. <ki...@di...>]

On 9 Aug 2007, at 05:00, Martin Flack wrote:

> For the type of thing like Connotea, Class::DBI is way too slow  
> left to its own devices - a page was taking 800 queries to pull up  
> all the pieces of data, and we solved that with a scary-looking but  
> fast-executing query. Another problem might be that it was designed  
> to be a very flexible query engine, and as it turns out that  
> flexibility is obscuring some of the functionality, i.e. tightening  
> the restraints might make prettier source code.

Oops,

I should mention that DBIx::Class has much better support for this  
kind of stuff.  It uses SQL::Abstract under the hood which really is  
an amazing piece of code once you get your head around it.  Passing  
raw sql into DBIC queries involves passing a scalar ref (i.e.  
\'select foo from bar') into the right part of the Resultset/SQLA  
code, but there's lots of other customisation available to make  
custom queries more convenient than this in many cases.

[Connotea-code-devel] More on dbix::class and social bookmarking code

[Kieren D. <ki...@di...>]

On 9 Aug 2007, at 05:00, Martin Flack wrote:

> For the type of thing like Connotea, Class::DBI is way too slow  
> left to its own devices - a page was taking 800 queries to pull up  
> all the pieces of data, and we solved that with a scary-looking but  
> fast-executing query. Another problem might be that it was designed  
> to be a very flexible query engine, and as it turns out that  
> flexibility is obscuring some of the functionality, i.e. tightening  
> the restraints might make prettier source code.

Oops, more on this.  One of my associates in Catalyst land has  
implemented a del.icio.us  clone in catalyst | dbic | tt | postgresql  
called linkmine.  It's main flaw is that it uses the "nice-idea- 
but..." library HTML::Widget - but it's most interesting from the  
database side. I've only looked at the database superficially, but  
the schema looks superficially similar (but simpler also).  With  
DBIC_TRACE=1 perl script/linkmine_server you can pretty much see what  
it's doing straight away.  The author is ZBY from CPAN (http:// 
search.cpan.org/~zby/ , http://perlalchemy.blogspot.com/)  and I'm  
sure he'd be amenable to either showing you the code or uploading the  
source to somewhere like sourceforge or google code if asked.

If you've got relationships declared in your database schema, getting  
something with the basic functions you need written to disk is as  
simple as this:

perl -MDBIx::Class::Schema::Loader=make_schema_at,dump_to_dir:/path/ 
to/lib -e  \
'make_schema_at("MySchemaName", {relationships => 1},  
["dbi:mysql:dbname=db", "user", "pass"])'

which by default will provide a good deal more than  
Class::DBI::Loader, and in a sane extendable manner.

Aside from this there's a ton of paid and community support for  
DBIx::Class (go to #dbix-class on irc.perl.org for details), and if  
there are bugs or missing features, it has a comprehnsive test suite,  
and commit to the svn is usually given out pretty freely after you've  
provided a couple of sane patches or tests.