Re: [Connotea-code-devel] feedback

[Martin F. <ma...@ne...>]

Hi Kieren,

Thanks for your email. I'm responsible for most of the codebase so I 
will step into the firing line first. ;-)

Kieren Diment wrote:
> I've been wanting to write this email for a while, and you can  
> consider me a strong supporter of what Connotea/NPG  is trying to  
> do.  However there are some major problems that I think need to be  
> addressed if connotea is to become a self sustaining open source  
> project with a sustainable developer community.

Again thanks. I'm aware of most of what you raise, and in fact have 
discussed them with folks, but it doesn't hurt to acknowledge on the 
devel list.

Actually this works out nicely since we are about to release a new 
version of Connotea Code literally any day now, and we are committed to 
posting a public darcs repository and inviting more public support. I'm 
actually thinking you have seen the new code by some of your comments.

> A bit about me:  I'm a social researcher and perl hacker.  Most of  
> the perl I write is for research data management, but I do  
> bibliographic stuff, web robots and occasional web applications  
> programming too.  I've been a significant contributor to the Catalyst  
> web application framework (mainly with the documentation and example  
> code).  I'm computer literate, and I've used mod_perl enough to have  
> a good idea of what irritates me about it.  I think that Catalyst is  
> the natural successor for 95% of the mod_perl or Apache::Registry  
> scripts out there.

I've looked at Catalyst but I've not used it.

> 1.  No test suite.  Lack of unit/behaviour tests along with the  
> impoverished debugging environment with mod_perl make for painful  
> development. [ SOLUTION:  Port to catalyst with built in testing  
> utilities and trivial support for perl -d ]

The new version of Connotea Code sports a fledgling test suite. It 
doesn't shoot for code coverage, just to test the basic functions. It's 
an area I'd like to strengthen.

I concur on the mod_perl debugging situation.

> 2.  Class::DBI + memcached.  These seem to be very tightly coupled.   
> Class::DBI is flawed software, and while it has proved very useful,  
> it clearly has severe limitations.  One example is the difficulty of  
> inspecting the sql that's it's generating.  It's main problem is a  
> general lack of transparency and implicitly generated code.  CDBI   
> "died" as an ongoing open source project about a year ago.  The  
> original author of CDBI's last project resulted in him fixing  
> DBIx::Class' CDBI Compatibility layer  so he could port his client's  
> code from CDBI  to DBIx::Class.

Accurate observation. At the time it seemed like the best way to provide 
abstractions that made the model possible.

I'd add a point that you didn't make, which is that Bibliotech::Query is 
  very complicated, not very clear, and the queries it generates are 
huge, but to some extent it's by necessity.

For the type of thing like Connotea, Class::DBI is way too slow left to 
its own devices - a page was taking 800 queries to pull up all the 
pieces of data, and we solved that with a scary-looking but 
fast-executing query. Another problem might be that it was designed to 
be a very flexible query engine, and as it turns out that flexibility is 
obscuring some of the functionality, i.e. tightening the restraints 
might make prettier source code.

What we got from Class::DBI was a convenient abstraction that database 
rows would be objects that could relate to each other. Beyond that we've 
extended the heck out of it, and probably not canonically, although in 
our defense Class::DBI is only designed to have a canonical usage for 
the theoretical database model, not a whole application.

The new version of Connotea Code has a Bibliotech::Query with even more 
speed optimizations but again it's another layer of complexity.

> [SOLUTION:  implied by the last comment DBIx::Class (DBIC)  is the  
> natural successor to CDBI, originally developed by the author of  
> CDBI::Sweet.  DBIC + memcached has production users.  Discussion on  
> IRC  (#dbix-class on irc.perl.org) indicates to me that swappable  
> caching engines (including a null cache for debugging) ) should be  
> trivial and transparent with a DBIC based data model.  The killer  
> dbic feature for me is that running your dbic script like so:   
> "DBIC_TRACE=1 perl run_my_script.pl" gives you the exact sql being  
> passed to the dbic classes.  This outputs to standard error, thus  
> making debugging hugely easier.  As noted, there's a CDBICompat layer  
> to ease transition from CDBI to DBIC.  CDBICompat has a more  
> extensive test suite than CDBI for good measure.  Finally DBIC tends  
> to result in much more efficent SQL than CDBI and swapping out  
> database engines (eg mysql to pg to sqlite) is much easier - I've  
> seen this done trivially from pg to sqlite for a database  
> representing a directed acyclic graph. ]

I've got SQL tracing - everything except do() calls and without 
question-mark translation - if you call 
Bibliotech::DBI->activate_warn_sql(). In mod_perl this sends up in 
Apache's error_log.

Again I've hard of DBIx::Class but was not aware that it offered any 
major benefits.

> 3.  Template toolkit.  The templates in connotea seem to be populated  
> with weird coderefs that make debugging/interrogation even more  
> difficult.  TT is one of the best templating solutions out there, but  
> connetea seems to misuse it. [SOLUTION:  Port to explicit templates  
> resident in files, using PROCESS, INCLUDE and MACRO blocks where  
> appropriate.  This should be fairly simple to do concurrently with  
> the rest of the catalyst/dbic port. ]

Anything that looks like misuse of TT is probably a result of yielding 
functionality from our original Component system to TT, where the rest 
of the Component system is still there, and which was more capable than 
what we perceived TT could do, and we needed to preserve functionality.

It may be that TT could do more than we realized, or could not then but 
can now, and in that case it would be, as you point out, better to use 
the native TT functionality rather than homegrown functionality. I agree 
on principle.

As an example, our Component system was designed to put all the logic 
for a component in one place, and avoid expensive recalculation. To that 
end, a component's result is not just a snippet of HTML like an INCLUDE, 
but an object that contains HTML parts that can be put in different 
places on the calling web page, as well as the Javascript for the <head> 
and Javascript for <body onload=""> that facilitate the component, so if 
you want to, say, position the cursor in the first form field, you can 
have the component return the HTML form and that Javascript together.

The new Connotea Code release has a section in the README to cover the 
TT function calls which should explain things better.

> 4.  Connotea doesn't scale down well.  I've used it on an ibook g4  
> for testing (performance verging on reasonable), and a pentium 3  
> linux machine with 128mb of RAM where performance was unacceptable.  
> [SOLUTON:  port to catalyst where fast_cgi, mod_perl and other more  
> exotic engines are all viable solutions.  Removing the mod_perl  
> depencency opens up shared hosting possibilities]

We used to have a bibliotech.cgi that could do exactly that (using 
Apache::Emulator) but we dropped it because NPG had no need for it and 
it was after all just a hack. I'd be happy to share it with you if you 
are interested, but the added memcached requirement is the main obstacle 
I think.

> There are some excellent things about connotea (auto-import and the  
> database schema being two big examples), but the above are show  
> stoppers which are going to cause huge problems for the  
> sustainability of the project.  The usability for end users is great,  
> but as far as programmer-usability goes connotea needs major  
> improvements otherwise it doesn't have a future as a viable open  
> source project.  Personally I'd like to hack in storage of pdf/other  
> fulltext into the database, but I can't do this in it's current  
> state.  I can also see why NPG won't do this themselves.

Thanks for the compliments!

At least you're not complaining it's not Ruby on Rails. ;-)

That's a joke, but I acknowledge that CPAN marches forwards even after 
we have selected our libraries and yes platforms make a huge difference. 
We selected solutions to match the engineering and then we tried to move 
to some newer approaches as dictated by practical concerns. Even moving 
to TT was only a direct response to needing non-programmers to do 
editing and design work.

> My vision is for distributed collaborative bibliographies, which is  
> why I'd like so much for the project to scale down to a level where  
> it could be used by 2-10 researchers on shared hosting.  I think  
> optional  re-import back to a master connotea  would be fairly easy  
> to implement after the scaling down problem was addressed.

Your work sounds interesting.

> Part of the problem is that connotea came about at a time of great  
> flux during the web app programming space, and the technology to  
> program these things has improved massively during this time.   

You are saying something that sounds to me like:

- refactoring for better library support would make easier development

I'd also add:

- like all things, more man-hours on the codebase for general 
refactoring or more test suite scripts would make easier development, 
keeping the same libraries even

- refactoring to remove some early assumptions would make a cleaner 
codebase as well

...and I'd argue that those two points are just as important.

> Unfortunately it leaves connotea with more technical debt.

Well, I think you may be being a bit hard on us here. ;-)

This is a classic challenge facing companies with working code; whether 
or not to spend time refactoring things.

When allocating sparse resources carefully, while being customer driven, 
a lot of the push is inevitably on keeping it running and adding 
features rather than revising the abstractions for something that 
already works.

Further, things that may make programming easier but don't actually 
lower the complexity level of the codebase are not as appealing in 
general as refactoring projects; e.g. switching from Class::DBI to 
DBIx::Class is not going to make it easier to understand, (maybe) just 
easier to work with. The concepts don't change much, you still have an 
abstraction layer that provides SQL support. You and I can appreciate 
that, but companies generally are interested in refactoring code when it 
can introduce junior programmers, not just make it easier for the senior 
programmers.

Having said that, your suggestions that we may be doing something in a 
non-standard manner, if they can be done in a standard manner, I do 
consider bugs to fix, as we should do things the same way as everyone 
else where possible.

If Class::DBI -> DBIx::Class is thought of as that type of problem, then 
I can appreciate the desire to switch over. But I'm speaking here more 
of things like using homegrown approaches over TT directives, as 
discussed above.

> I'm happy to clarify anything I've written here on request.  If the  
> response is going to be "isn't going to happen, sorry about that",  
> I'm prepared to accept that too.  As a first step I'd recommend  
> popping on to #catalyst at irc.perl.org and asking about porting  
> mod_perl apps to Catalyst.

I'm not the final word, and even if I was, what you've presented 
deserves study and thought before issuing the final word. ;-)

I suggest we keep the conversation going.

Regards,
Martin Flack