gusdev-gusdev

[GUSDEV] out of memory when loading taxonomy

[Jian Lu <jl...@vb...>]

Hi GUSDEV,

When I was using GUS plug-in to load NCBI taxonomy, it failed. The error 
message is 'Out of Memory'. My machine's RAM is 4G which should be 
enough to hold the data. Any hint would be appreciated. Thanks,

Jian

Re: [GUSDEV] out of memory when loading taxonomy

[Fernan A. <fe...@ii...>]

+----[ Jian Lu <jl...@vb...> (31.May.2006 10:45):
|
| Hi GUSDEV,
| 
| When I was using GUS plug-in to load NCBI taxonomy, it failed. The error 
| message is 'Out of Memory'. My machine's RAM is 4G which should be 
| enough to hold the data. Any hint would be appreciated. Thanks,
| 
| Jian
|
+----]

Jian,

can you post the exact error message (copy & paste)?

Is it Perl running out of memory? Or is it the db
(postgres/oracle)?

Your box has 4Gb RAM, but this doesn't mean that all of this
memory is available to the application.

Are you running the plugin from one host and the db server
in another host? If so, which one is 'getting out of
memory'?


What RDBMS are you using? Postgres? Oracle?

Have you set up your RDBMS to use the extra available
memory? Both Oracle and Postgres should be configured to use
more memory, if it's available. The default configuration
that (the one you get from a fresh install) would give you a
horrid performance and will not make use of extra RAM.


What OS are you using? Linux? FreeBSD? 

Have you tuned the OS? (through building a new kernel or
modifying kernel tunables using sysctl or something
similar?) Raising maxusers, maxfiles and/or a number of
kernel parameters can give your app more room to scale if
necessary. 


Is this a dedicated db server or is it running other
applications alongside postgres or oracle? Even if you tune
your db and OS to maximize the use of resources, running two
or more apps that compete for the same resource will lead to
degraded performance.

As you can see, there are a number of steps that you should
follow to make use of the available RAM and to optimize
performance of your DB app. 

And we need more information about your setup to be able to
guess what the problem might be.

Fernan

PS: I've succesfully loaded NCBI Taxonomy (in ~ 2hs) on
PostgreSQL running on a spare box that was not particularly
beefy (AMD Sempron, 1 Gb RAM, RAID-1 ATA100). But both the OS
(FreeBSD) and the application (PostgresSQL) were tuned to
maximize use of available resources.

Re: [GUSDEV] out of memory when loading taxonomy

[Jinal J. <jaj...@lb...>]

Hi Jian,

What are your ulimit settings? You can get it by typing "ulimit -a"



Thanks
--Jinal

On 5/31/06, Fernan Aguero <fe...@ii...> wrote:
>
> +----[ Jian Lu <jl...@vb...> (31.May.2006 10:45):
> |
> | Hi GUSDEV,
> |
> | When I was using GUS plug-in to load NCBI taxonomy, it failed. The error
> | message is 'Out of Memory'. My machine's RAM is 4G which should be
> | enough to hold the data. Any hint would be appreciated. Thanks,
> |
> | Jian
> |
> +----]
>
> Jian,
>
> can you post the exact error message (copy & paste)?
>
> Is it Perl running out of memory? Or is it the db
> (postgres/oracle)?
>
> Your box has 4Gb RAM, but this doesn't mean that all of this
> memory is available to the application.
>
> Are you running the plugin from one host and the db server
> in another host? If so, which one is 'getting out of
> memory'?
>
>
> What RDBMS are you using? Postgres? Oracle?
>
> Have you set up your RDBMS to use the extra available
> memory? Both Oracle and Postgres should be configured to use
> more memory, if it's available. The default configuration
> that (the one you get from a fresh install) would give you a
> horrid performance and will not make use of extra RAM.
>
>
> What OS are you using? Linux? FreeBSD?
>
> Have you tuned the OS? (through building a new kernel or
> modifying kernel tunables using sysctl or something
> similar?) Raising maxusers, maxfiles and/or a number of
> kernel parameters can give your app more room to scale if
> necessary.
>
>
> Is this a dedicated db server or is it running other
> applications alongside postgres or oracle? Even if you tune
> your db and OS to maximize the use of resources, running two
> or more apps that compete for the same resource will lead to
> degraded performance.
>
> As you can see, there are a number of steps that you should
> follow to make use of the available RAM and to optimize
> performance of your DB app.
>
> And we need more information about your setup to be able to
> guess what the problem might be.
>
> Fernan
>
> PS: I've succesfully loaded NCBI Taxonomy (in ~ 2hs) on
> PostgreSQL running on a spare box that was not particularly
> beefy (AMD Sempron, 1 Gb RAM, RAID-1 ATA100). But both the OS
> (FreeBSD) and the application (PostgresSQL) were tuned to
> maximize use of available resources.
>
>
> -------------------------------------------------------
> All the advantages of Linux Managed Hosting--Without the Cost and Risk!
> Fully trained technicians. The highest number of Red Hat certifications in
> the hosting industry. Fanatical Support. Click to learn more
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642
> _______________________________________________
> Gusdev-gusdev mailing list
> Gus...@li...
> https://lists.sourceforge.net/lists/listinfo/gusdev-gusdev
>

Re: [GUSDEV] out of memory when loading taxonomy

[Jian Lu <jl...@vb...>]

Thanks for such detail guidance.

Here is the error message output:

Out of memory!
Issuing rollback() for database handle being DESTROY'd without explicit 
disconnect(), <MERGED> line 5401.
Issuing rollback() for database handle being DESTROY'd without explicit 
disconnect(), <MERGED> line 5401.

Database: Oracle 9i on Sun Solaris
Application: SuSE Linux Enterprise Server 9

By running ulimit -a, the output is 
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 27636
virtual memory        (kbytes, -v) unlimited


My confusion is I used the same application to successfully load 
taxonomy to another database instance located on the same RDBMS machine. 
I didn't run them at the same time.

Fernan Aguero wrote:
> +----[ Jian Lu <jl...@vb...> (31.May.2006 10:45):
> |
> | Hi GUSDEV,
> | 
> | When I was using GUS plug-in to load NCBI taxonomy, it failed. The error 
> | message is 'Out of Memory'. My machine's RAM is 4G which should be 
> | enough to hold the data. Any hint would be appreciated. Thanks,
> | 
> | Jian
> |
> +----]
>
> Jian,
>
> can you post the exact error message (copy & paste)?
>
> Is it Perl running out of memory? Or is it the db
> (postgres/oracle)?
>
> Your box has 4Gb RAM, but this doesn't mean that all of this
> memory is available to the application.
>
> Are you running the plugin from one host and the db server
> in another host? If so, which one is 'getting out of
> memory'?
>
>
> What RDBMS are you using? Postgres? Oracle?
>
> Have you set up your RDBMS to use the extra available
> memory? Both Oracle and Postgres should be configured to use
> more memory, if it's available. The default configuration
> that (the one you get from a fresh install) would give you a
> horrid performance and will not make use of extra RAM.
>
>
> What OS are you using? Linux? FreeBSD? 
>
> Have you tuned the OS? (through building a new kernel or
> modifying kernel tunables using sysctl or something
> similar?) Raising maxusers, maxfiles and/or a number of
> kernel parameters can give your app more room to scale if
> necessary. 
>
>
> Is this a dedicated db server or is it running other
> applications alongside postgres or oracle? Even if you tune
> your db and OS to maximize the use of resources, running two
> or more apps that compete for the same resource will lead to
> degraded performance.
>
> As you can see, there are a number of steps that you should
> follow to make use of the available RAM and to optimize
> performance of your DB app. 
>
> And we need more information about your setup to be able to
> guess what the problem might be.
>
> Fernan
>
> PS: I've succesfully loaded NCBI Taxonomy (in ~ 2hs) on
> PostgreSQL running on a spare box that was not particularly
> beefy (AMD Sempron, 1 Gb RAM, RAID-1 ATA100). But both the OS
> (FreeBSD) and the application (PostgresSQL) were tuned to
> maximize use of available resources.
>