Re: [Connotea-code-devel] MVC.

[Martin F. <ma...@ne...>]

Kieren, thanks for these comments.

The bulk of our work is on citation/import/export, speed optimization, 
user features, etc. so I am not averse at all to having some of the 
structural points reexamined, and you're probably correct that we 
haven't thought about them questioningly in a long time. If this is an 
issue for newcomers I'd like to be aware of it.

Kieren Diment wrote:
> 1.  Getting the database code working independently of the web  
> application and calling it a well separated model.

Your WIAB project looks file-based; I'd be interested in a similarly 
small SQL project.

> 2.  Taking your dispatcher code, getting it working independent of  
> apache (basically factoring out $r into utility functions).  Next up,  
> delegating the dispatching logic and content generation logic below  
> and bending it into catalyst logic.  I really think this isn't going  
> hard for someone familiar with the codebase, although it would be  
> tricky for me.  What this buys you is a clearly understood dispatch  
> process that's very easily understood by a large and active developer  
> community.

$r is basically only handled in Apache.pm. You can write a test script 
that creates a Bibliotech::Command or uses Bibliotech::Parser to create 
one. Bibliotech::Fake was created for this purpose.

Having said that, you are correct it would be good to add more 
abstraction to make this easier, and make command-line uses easier.

I suppose at the moment I am perhaps so familiar with the codebase I can 
write a test script "too quickly."

>> I think you'd get your CMV separation as good as any application  
>> can get with these steps:
>>
>> - removing the *_content() methods and supporting items and making  
>> them subclasses of the Bibliotech::Page and Bibliotech::Component  
>> objects with a content() method. That just seemed complicated
>>
>> - removing the *_content() methods and supporting items from  
>> Bibliotech::DBI objects and putting them into similarly separated  
>> subclasses or have another class that accepts objects and renders  
>> them or something like that.
>>
>> It just seemed like that increased syntactical complexity and class  
>> complexity in order to create separation that we didn't think we  
>> needed when the codebase was smaller.
>>
> 
> mmm.  I really think that's called technical debt

Perhaps an explicit CMV remodel like this will turn out to be the first 
suggestion offered by you that we would try to implement then.

>> I'm actually interested in seeing a fairly big application written  
>> in Catalyst to see how you guys do it.
> 
> right.  The really big ones tend to be closed source.  A prominent  
> one ( Vox from 6apart):  http://tokyo2007.yapcasia.org/sessions/ 
> 2007/02/everything_vox.html

I certainly will take a look when I have time.

> and a little experiment:
> 
> /tmp$ du -h -d 1 connotea-code-1.7.1
> 1.0M    connotea-code-1.7.1/bibliotech
> 1.0M    connotea-code-1.7.1

I don't understand what your actual point here is though.

> Here's my Website in a Box which I wrote while learning to provide a  
> complete and useful calalyst app with good test coverage and good  
> programmer documentation.  The dispatch logic here is very simple.   
> http://code.google.com/p/websiteinabox (codebase = 1.3M).  It has  
> some flaws (the controller is too thick in places and the model too  
> thin mainly).

Interesting.

> Now onto the Model:
> 
>> Unfortunately we are not even within the bounds of Class::DBI  
>> because Bibliotech::Query does a lot of query building itself,  
>> although I am interested in revamping it because it is the most  
>> difficult part of the codebase. So we'd probably have to do a  
>> general refactoring before we even switched.
>>
> 
> Right.   DBIC was made because projects outgrew CDBI very quickly in  
> exactly this manner. In this case I'd really recommend doing a  
> concurrent refactor to DBIx::Class to get a model that works  
> independently of the app.  Getting a starting schema with good  
> relationships coverage really is as simple as installing DBIx::Class  
> and doing something like:

I'm not convinced your examples are as aggressive as we are in the SQL 
area. Just for giggles, I'm going to post a query that collects the data 
for /user/martin while logged in as martin. This will give you an idea 
of the consolidation that was necessary to collect the data for one page 
of Connotea in a reasonable number of seconds. A similar set of queries 
to collect all the requisite data was over 800 queries (really) when 
leaving Class::DBI to get the data itself so this is what it looks like 
now after packing it into one monster query that runs quickly. I'll post 
in a separate email.

I'm not saying you're wrong, just take a look if you like.

> You've got a lot of sql you can then port back in by using the scalar  
> ref feature in SQL::Abstract, and from there you can wind back to  
> something a bit easier to maintain using core dbic functionality  
> wherever possible.

BTW, we did use SQL::Abstract, so we do know about it.

> chainable resultsets are a feature everybody finds extremely useful:
> 
> my $schema  = DB->connect();
>   my $resultset = $schema->resultset('Foo')->search(\% 
> scary_search_condition, \%search_attributes);
> my $narrowed_resultset = $resultset->search(\%some_more_search_logic);
> while (my $foo = $narrowed_resultset->next) { # you only hit the  
> database at this point in the chaining of the resultset }

Ok. I'll have to study that class.

> things like using a cache engine ie memcached should be transparent  
> and trivial to remove for users who don't need that feature with DBIC  
> (c.f.  tight couplling and  real maintenance headache for the  
> equivalent CDBI code).

It is abstracted in Bibliotech::Cache so maybe just a 
Bibliotech::Cache::None or Bibliotech::Cache::Memory gets added to the 
project? A little bit of the caching is useful in the same request.

Do you happen to know of any public Catalyst projects that are required 
to use server-wide caching to handle the traffic load? So we could see 
their programming model for the cache interface? That would be interesting.

I'd note we cache in M and V stages in Connotea. Query results as 
constituted objects as well as component HTML and whole pages (HTML or 
otherwise) are cached, for visitors, and for individuals (since privacy 
affects everything we show in Connotea).

> DBIx:Class ought to see the following schema from the connotea db  by  
> default (ie relationships pre-specified for you in the generated  
> code) - http://www.uow.edu.au/~kdiment/schema.png (generated by  
> sqlfairy which DBIC uses for some purposes). I can't guarantee this,  
> but if not it will be close.

Sure.

Cheers,
Martin