I have had problems with running out of memory and virtual memory with really large MySQL tables that have been nicely resolved by using --quick with mysqldump.
--quick - This option is useful for dumping large tables. It forces mysqldump to retrieve rows for a table from the server a row at a time rather than retrieving the entire row set and buffering it in memory before writing it out...
2007-06-12 01:52:03 UTC in Webmin