From: Jon P. <jo...@cr...> - 2007-03-21 01:59:27
|
On Sat, 2007-03-17 at 13:02 -0700, Victor Stone wrote: > On 3/16/07, Mike Linksvayer <ml...@cr...> wrote: <snip /> > The Query API currently washes out potential privacy issues, both user > data and internal paths so that's there, but doing anything through > our current Query API or even feed code does not scale at all past a > few dozen records at a time (ccM is near or at 6,000 uploads). > > iow, the current datadump code is worthless at our scale, I doubt it works. Yes, it seems to be working if the memory_limit php setting is high enough to deal with the load. I added a ini_set for this CLI option to the data dump code. It appears to be working fine on ccmixter.org. I added this for openclipart.org's datadump to work. That is ok temporarily, but another approach should be found for scalability, soon... > new, memory and sql efficient code, probably ignoring phptal templates > would probably have to be written to do this right, which is fine > because I would think csv is better for doing data mining than xml. > I'd cost it at a week before the kinks are worked out. > <snip /> -- Jon Phillips jo...@cr... cell: 510.499.0894 Community/Business Developer Creative Commons www.creativecommons.org |