Re: [GUSDEV] out of memory when loading taxonomy

[Jian Lu <jl...@vb...>]

Thanks for such detail guidance.

Here is the error message output:

Out of memory!
Issuing rollback() for database handle being DESTROY'd without explicit 
disconnect(), <MERGED> line 5401.
Issuing rollback() for database handle being DESTROY'd without explicit 
disconnect(), <MERGED> line 5401.

Database: Oracle 9i on Sun Solaris
Application: SuSE Linux Enterprise Server 9

By running ulimit -a, the output is 
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 27636
virtual memory        (kbytes, -v) unlimited


My confusion is I used the same application to successfully load 
taxonomy to another database instance located on the same RDBMS machine. 
I didn't run them at the same time.

Fernan Aguero wrote:
> +----[ Jian Lu <jl...@vb...> (31.May.2006 10:45):
> |
> | Hi GUSDEV,
> | 
> | When I was using GUS plug-in to load NCBI taxonomy, it failed. The error 
> | message is 'Out of Memory'. My machine's RAM is 4G which should be 
> | enough to hold the data. Any hint would be appreciated. Thanks,
> | 
> | Jian
> |
> +----]
>
> Jian,
>
> can you post the exact error message (copy & paste)?
>
> Is it Perl running out of memory? Or is it the db
> (postgres/oracle)?
>
> Your box has 4Gb RAM, but this doesn't mean that all of this
> memory is available to the application.
>
> Are you running the plugin from one host and the db server
> in another host? If so, which one is 'getting out of
> memory'?
>
>
> What RDBMS are you using? Postgres? Oracle?
>
> Have you set up your RDBMS to use the extra available
> memory? Both Oracle and Postgres should be configured to use
> more memory, if it's available. The default configuration
> that (the one you get from a fresh install) would give you a
> horrid performance and will not make use of extra RAM.
>
>
> What OS are you using? Linux? FreeBSD? 
>
> Have you tuned the OS? (through building a new kernel or
> modifying kernel tunables using sysctl or something
> similar?) Raising maxusers, maxfiles and/or a number of
> kernel parameters can give your app more room to scale if
> necessary. 
>
>
> Is this a dedicated db server or is it running other
> applications alongside postgres or oracle? Even if you tune
> your db and OS to maximize the use of resources, running two
> or more apps that compete for the same resource will lead to
> degraded performance.
>
> As you can see, there are a number of steps that you should
> follow to make use of the available RAM and to optimize
> performance of your DB app. 
>
> And we need more information about your setup to be able to
> guess what the problem might be.
>
> Fernan
>
> PS: I've succesfully loaded NCBI Taxonomy (in ~ 2hs) on
> PostgreSQL running on a spare box that was not particularly
> beefy (AMD Sempron, 1 Gb RAM, RAID-1 ATA100). But both the OS
> (FreeBSD) and the application (PostgresSQL) were tuned to
> maximize use of available resources.
>