Menu ▾ ▴

#226 Inode problem apache + htdig-3.2.0.b5

open
nobody
htdig (103)
7
2004-06-09
2004-06-09
Matthew
No

Good day fokes,

I`ve upgraded htdig from 3.1.6 to 3.2.0.b5.
Some how since the upgrade the htdig program isn`t operating how
it used to be.

- Its running much slower.
- It creates a tremendous load on apache

20347 www 59 0 34120K 23152K RUN 0:21 42.63%
42.63% httpd
20345 www 2 0 33364K 22512K sbwait 1:00 23.97%
23.97% httpd
20729 www 58 0 32600K 21552K RUN 0:05 43.14%
19.48% httpd
20642 www 18 0 32600K 21584K lockf 0:30 4.50%
4.49% httpd

- It causes apache to keep its allocated inodes during the run of htdig.

After giving apache a graceful or restart it will give back the inodes,
but when not doing so it will keep the inodes and causes the hd to
run full.
But in effect there is no such file or directory that takes the
diskspace, its apache still keeping the inodes allocated.

I was wondering if any of you people are experiencing the same
problem.

I am running the following platform:
FreeBSD 4.9-RELEASE-p4
apache+mod_ssl-1.3.31+2.8.17_3
mod_php4-4.3.6_1,1
mod_perl-1.29
perl-5.8.2_5

If any other information is needed just give me a shout and i`ll comply.

With kind regards

Matthew Whittaker-Williams

Discussion

  • Lachlan Andrew

    Lachlan Andrew - 2004-06-09

    Logged In: YES
    user_id=663373

    Greetings,

    Yes, we know that 3.2.0 is much slower than 3.1.6.
    However, we (I) didn't know that it also uses a lot more
    resources from Apache. That will help us a lot in
    tracking down the problem.

    Could you please see what happens if you turn off the
    "head before get" option?

    Thanks for the bug report,
    Lachlan

     

  • Lachlan Andrew

    Lachlan Andrew - 2004-06-09
    • priority: 5 --> 7
     

  • Matthew

    Matthew - 2004-06-09

    Logged In: YES
    user_id=1059959

    After setting the value head_before_get option in my htdig config:

    Begin of htdig run:

    Filesystem Size Used Avail Capacity Mounted on
    /dev/da0s1a 252M 55M 176M 24% /
    /dev/da0s1h 28G 7.7G 18G 30% /home
    /dev/da0s1e 252M 73M 159M 32% /tmp
    /dev/da0s1g 3.9G 1.6G 2.0G 45% /usr
    /dev/da0s1f 1008M 688M 239M 74% /var
    procfs 4.0K 4.0K 0B 100% /proc
    --
    Filesystem Size Used Avail Capacity Mounted on
    /dev/da0s1a 252M 55M 176M 24% /
    /dev/da0s1h 28G 7.9G 18G 31% /home
    /dev/da0s1e 252M 73M 159M 32% /tmp
    /dev/da0s1g 3.9G 1.6G 2.0G 45% /usr
    /dev/da0s1f 1008M 688M 240M 74% /var
    procfs 4.0K 4.0K 0B 100% /proc
    --
    Filesystem Size Used Avail Capacity Mounted on
    /dev/da0s1a 252M 55M 176M 24% /
    /dev/da0s1h 28G 9.0G 17G 35% /home
    /dev/da0s1e 252M 73M 159M 32% /tmp
    /dev/da0s1g 3.9G 1.6G 2.0G 45% /usr
    /dev/da0s1f 1008M 688M 240M 74% /var
    procfs 4.0K 4.0K 0B 100% /proc
    --
    Filesystem Size Used Avail Capacity Mounted on
    /dev/da0s1a 252M 55M 176M 24% /
    /dev/da0s1h 28G 11G 15G 43% /home
    /dev/da0s1e 252M 73M 159M 32% /tmp
    /dev/da0s1g 3.9G 1.6G 2.0G 45% /usr
    /dev/da0s1f 1008M 688M 240M 74% /var
    procfs 4.0K 4.0K 0B 100% /proc

    --
    28729 www 63 0 34320K 23836K RUN 0:24 21.00% 21.00%
    httpd
    27770 www 62 0 35772K 25380K RUN 0:19 20.85% 20.85%
    httpd
    28769 www 63 0 34412K 23920K RUN 0:43 19.58% 19.58%
    httpd
    29022 www 62 0 33788K 23044K RUN 0:29 19.39% 19.38%
    httpd
    27764 www 2 0 35300K 24888K sbwait 0:31 5.42% 5.42%
    httpd
    27783 www 2 0 35928K 25532K sbwait 0:31 1.22% 1.22%
    httpd
    29106 www 2 0 34412K 23844K sbwait 0:00 0.12% 0.05%
    httpd
    --

    finished htdig run:

    Filesystem Size Used Avail Capacity Mounted on
    /dev/da0s1a 252M 55M 176M 24% /
    /dev/da0s1h 28G 12G 13G 49% /home
    /dev/da0s1e 252M 73M 159M 32% /tmp
    /dev/da0s1g 3.9G 1.6G 2.0G 45% /usr
    /dev/da0s1f 1008M 688M 240M 74% /var
    procfs 4.0K 4.0K 0B 100% /proc
    --
    27770 www 2 0 35772K 25396K sbwait 0:31 0.34% 0.34%
    httpd
    27783 www 2 0 35744K 25340K sbwait 0:31 0.10% 0.10%
    httpd
    29161 www 2 0 34384K 23848K sbwait 0:00 0.05% 0.05%
    httpd
    85828 root 2 0 33788K 22660K select 1:20 0.00% 0.00%
    httpd
    27777 www 2 0 36316K 25844K sbwait 0:31 0.00% 0.00%
    httpd
    27782 www 2 0 35620K 25184K sbwait 0:31 0.00% 0.00%
    httpd
    27764 www 2 0 35300K 24888K sbwait 0:31 0.00% 0.00%
    httpd
    27772 www 2 0 36416K 26004K sbwait 0:31 0.00% 0.00%
    httpd
    28858 www 2 0 34480K 23928K sbwait 0:30 0.00% 0.00%
    httpd

    After reloading Apache:

    Filesystem Size Used Avail Capacity Mounted on
    /dev/da0s1a 252M 55M 176M 24% /
    /dev/da0s1h 28G 12G 13G 49% /home
    /dev/da0s1e 252M 73M 159M 32% /tmp
    /dev/da0s1g 3.9G 1.6G 2.0G 45% /usr
    /dev/da0s1f 1008M 688M 240M 74% /var
    procfs 4.0K 4.0K 0B 100% /proc

    It finaly hits me why apache took so many inodes, this is because a site the
    htdig engine is going trough has large errors on the site causing the error
    log to fill up, while apache preserves its allocated memory and doesn`t let it
    go.
    I normally rotate the log file once awhile in newsyslogd cause it was previous
    causing problems on the server.
    But after disabling the option head_before_get the file just get written and
    takes 5.2 gig on hd , before that option was on by default it didn`t show
    that the log file was that large ( Apache just didn`t show it ).

    So i guess that is not really a bug but a small problem.
    Apache load went down using head_before_get option disabled.
    So i guess there is still some work on 3.2.0...

    Well goodluck & Thanks

    Kind regards

    Matthew

     

Log in to post a comment.