fofredux-devel

[Fofredux-devel] Reduce cpu usage?

[Miles B. <mil...@gm...>]

Hello,

I've run into a problem using FoFRedux on a shared host. A few times I
have had my account suspended briefly due to excessive cpu
utilization. The error that I get is that the account had used more
than 20% of the cpu and the account has briefly been suspended.

Is there a way to still have FoFREdux work but limit the cpu utilization?

I have 48 feeds that are checked once per hour.

Thanks,

-Miles

Re: [Fofredux-devel] Reduce cpu usage?

[Miles B. <mil...@gm...>]

Here is the log of the cpu utlization. Is there a way to find out
which feed is taking so long?

--------------------------------------------------------------------------------
CPU Exceeded Log For Thu Oct 26 11:30:55 2006

  Thu Oct 26 11:30:30 2006: used  0.19 seconds of cpu time for HTTP
Request: : GET /?feed=rss2 HTTP/1.0
  Thu Oct 26 11:30:50 2006: used  41.00 seconds of cpu time for [[mysql query]]
--------------------------------------------------------------------------------
CPU Exceeded Log For Thu Oct 26 11:31:16 2006

  Thu Oct 26 11:30:30 2006: used  0.19 seconds of cpu time for HTTP
Request: : GET /?feed=rss2 HTTP/1.0
  Thu Oct 26 11:30:50 2006: used  41.00 seconds of cpu time for [[mysql query]]
--------------------------------------------------------------------------------
CPU Exceeded Log For Thu Oct 26 11:31:32 2006

  Thu Oct 26 11:30:30 2006: used  0.19 seconds of cpu time for HTTP
Request: : GET /?feed=rss2 HTTP/1.0
  Thu Oct 26 11:30:50 2006: used  41.00 seconds of cpu time for [[mysql query]]

On 1/31/07, Evan Roth <eva...@gm...> wrote:
> i would start by checking the feeds more intermittently.
> you can set a cron for each feed to:
> http://yourhost/fofinstall/update.php?feed=XX
> (see the link if you mouseover the u (for update) in the frameview)

Thanks for the suggestion on spacing out the cron jobs. But having 48
seperate cron jobs kinda is overkill.

I did notice that when you click on the Edit link for a feed you can
set an update interval. I tried setting a feed to a one minute update
and it did not seem to work. Using this method (if it worked) I could
set up half my feeds to be every 30 minutes, and the other half to be
every 60 minutes. Though even this would not solve the problem as it
is an excessive mysql query which seems to be the culprit.

Re: [Fofredux-devel] Reduce cpu usage?

[Andrew T. <ajt...@hi...>]

On 1/31/07, Miles Beck <mil...@gm...> wrote:
> Here is the log of the cpu utlization. Is there a way to find out
> I did notice that when you click on the Edit link for a feed you can
> set an update interval. I tried setting a feed to a one minute update
> and it did not seem to work. Using this method (if it worked) I could
> set up half my feeds to be every 30 minutes, and the other half to be
> every 60 minutes. Though even this would not solve the problem as it
> is an excessive mysql query which seems to be the culprit.

Yeah, I don't know when the 'update interval' showed up, but I don't
believe it's being used currently. The 'slickest'/easiest solution
would be to included a PHP script (or just bash script) that would
handle spacing out updating your feeds.

It would be called by cron every 30 or 60 mins, then do a query to the
DB to get, say, the 5 oldest feeds that are due for updating (based on
update interval) and update just those, and the next time cron called
it, it would get 5 (or whatever) different ones.

Sounds like something Kevin could probably do in 10 mins ;)

Re: [Fofredux-devel] Reduce cpu usage?

[Kevin <ke...@dr...>]

> On 1/31/07, Miles Beck <mil...@gm...> wrote:
>> Here is the log of the cpu utlization. Is there a way to find out
>> I did notice that when you click on the Edit link for a feed you can
>> set an update interval. I tried setting a feed to a one minute update
>> and it did not seem to work. Using this method (if it worked) I could
>> set up half my feeds to be every 30 minutes, and the other half to be
>> every 60 minutes. Though even this would not solve the problem as it
>> is an excessive mysql query which seems to be the culprit.
>
> Yeah, I don't know when the 'update interval' showed up, but I don't
> believe it's being used currently. The 'slickest'/easiest solution
> would be to included a PHP script (or just bash script) that would
> handle spacing out updating your feeds.
>
> It would be called by cron every 30 or 60 mins, then do a query to the
> DB to get, say, the 5 oldest feeds that are due for updating (based on
> update interval) and update just those, and the next time cron called
> it, it would get 5 (or whatever) different ones.

Variable feed updates has been a feature for a while.

1) enable variable feed updates in settings page.
2) define an update interval for all your feeds, based on how fresh you
want the feed to be.  (default is 60min)
3) run update-quiet.php from cron periodically. (I do it every 30 mins)

What it does is only update a feed if the last time it was updated is
longer then $UPDATE_INTERVAL for that feed.  So every time
update-quiet.php is run, only a small number of feeds are updated.

For example: A feed like a web comic that only changes once a day, can
have an update interval set much higher (several hours) compared to a
chatty blog or news website.

The user can adjust the update intervals to their own liking.

--=20
Kevin

Re: [Fofredux-devel] Reduce cpu usage?

[Miles B. <mil...@gm...>]

This looks like the time I had set my feeds to be updated:

Wed Jan 31 09:59:10 2007: used  0.34 seconds of cpu time for HTTP
Request: : GET /?feed=rss2 HTTP/1.1

The others are from the blog that is hosted on the machine that my
wife writes. Looks like the rss feeds to the blog articles took a long
time for some reason.

On 1/31/07, Evan Roth <eva...@gm...> wrote:
> are you positive the problem is occurring with an update of feeds...and not
> simply a get, when you are reading?
>
> you could also try to put your update at a weird time, like 19 minutes after
> the hour.  i could imagine that lots of people have crons running on the
> hour...which would possibly add up to a cumulative delay for all the jobs.

That's a good idea. I'll try that as well and monitor the cpu_exceeded
logs to see when/if this occurs again.

Re: [Fofredux-devel] Reduce cpu usage?

[Miles B. <mil...@gm...>]

I do have access to phpmysladmin. I did an optimization and here are
the differences.

The total table size went from 14.5 MiB to 13.2 MiB.
The overhead went from 736.3 KiB to 0 KiB.

-Miles

On 1/31/07, Evan Roth <eva...@gm...> wrote:
> if you have phpmysqladmin access to your database, try to optimize all of
> the tables.  with 48 feeds, i could imagine that the items table could be
> quite fragmented, if you are not reading often and deleting stuff as it gets
> added.  perhaps your provider would let you see the SQL that is taking so long on
> the server, to allow us to optimize.  we could also take a deeper look at
> the db itself...perhaps we should add some indexing somewhere or something.
> or at least it tells us exactly what is causing the problem.

Re: [Fofredux-devel] Reduce cpu usage?

[Miles B. <mil...@gm...>]

On 1/31/07, Kevin <ke...@dr...> wrote:
>
>   I forget, what snapshot are you running?

It looks like I have the snapshot right before the index change was
made. A lot of the files have a time/date stamp of July of 2006 and a
few of the directories have a date of August 3rd. So I could have the
changes you made, or I just missed getting the changes.

It sounds like Wordpress is having issues. I checked the biggest log
and it showed 25 entries that took too long. One which used 42 seconds
of cpu time and all the others from .20 to .71 seconds. It looks like
the combined GETs that were slow was the culprit.

Thanks for the help.

-Miles