Re: [Cctools-cchost] sources in feeds and db access for researchers

[Jon P. <jo...@cr...>]

On Wed, 2007-03-21 at 08:32 -0800, Victor Stone wrote:
> On 3/20/07, Jon Phillips <jo...@cr...> wrote:
> > On Tue, 2007-03-20 at 21:28 -0700, Jon Phillips wrote:
> > > On Tue, 2007-03-20 at 19:04 -0700, Mike Linksvayer wrote:
> > > > On Tue, 2007-03-20 at 18:59 -0700, Jon Phillips wrote:
> > > > > Yes, it seems to be working if the memory_limit php setting is high
> > > > > enough to deal with the load. I added a ini_set for this CLI option to
> > > > > the data dump code. It appears to be working fine on ccmixter.org. I
> > > > > added this for openclipart.org's datadump to work.
> > > > >
> > > > > That is ok temporarily, but another approach should be found for
> > > > > scalability, soon...
> 
> and I don't think that even works if you don't have root access on the
> box in question (shared hosters don't like it when one client boosts
> the memory limit)

No, it works depending on configuration of php setup...I tested on my
box, dreamhost and a4, and all seem to work.

> > > >
> > > > Is the entire dump being constructed in memory before being written?  Or
> > > > the entire db resultset read into memory before the dump is constructed?
> > >
> > > The first one unfortunately...Yes, it needs to be paged through and
> > > written piece by piece...
> >
> > I haven't looked at the new query code too much...VS, what is best way
> > to accomplish this now with paging and new query code?
> > > > here's no reason a dump should be memory limited.  If we're using a
> > > > library that does one or both of the things above it is bogus. :-)
> > >
> > > Yah, agree...one of those fastest line to getting done tasks if I
> > > remember ;) I'll add it to my list ;)
> 
> the query and feed code was not designed to handle a case with 6,000
> records, typical feed is 15 items, typical number of records shown on
> a web page is about the same.
> 
> as I said in earlier message on this thread, new code, probably not
> even using the phptal template system and using raw, unadorned mysql
> calls should be written to do this kind of dump. I'm totally swamped
> by a bad cold but I can do this when I'm thinking straight.

Bummer...get betta soon...

Jon

-- 
Jon Phillips
jo...@cr...
cell: 510.499.0894

Community/Business Developer
Creative Commons
www.creativecommons.org