Whats the cause for this error, and how can I fix it?
System is on Debian Etch, trying to fix a 1.5 TB Ext3 Partition of a Raid 5 on a 3ware 9550SXU Controller.
2 GB RAM + 2 GB Swap available.
If you need any further information i'll supply it happily, just say so.
Thanks,
Christian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You're probably using something like BackupPC or or pdumpfs which creates a very large number of hard link trees. This exacerbates two memory usage problems:
* Even if directories are equivalent, Unix does not allow directories to
be hardlinked, so if a filesystem has 100,000 directories, each
incremental backup will create 100,000 new directories in the BackupPC
directory. E2fsck requires 12 bytes of storage per directory in order
to store accounting information.
* E2fsck uses an icount abstraction to store the i_links_count
information from the inode, as well as the number of times an inode is
actually referenced by directory. This abstraction uses an
optimization based on the observation that on most normal filesystems,
there are very few hard links (i.e., i_links_count for most regular
files is 1). The icount abstraction uses 6 bytes of memory for each
directory and regular file which has been hardlinked, and two of them
are used.
One such filesystem that was reported to me had 88 million inodes, of which 11 million were files, and 77 million were directories (!). This meant that e2fsck needed to allocate around 881 megabytes of memory in one contiguous array for the dirinfo data structures, and two (approximately) 500 megabyte contiguous arrays for the icount abstraction.
On a 32-bit processor, especially with shared libraries enabled to futher reduce the amount of available 3GB address space, e2fsck can very easily fail to have enough memory. Using a statically-linked e2fsck can help, as can moving to a 64-bit processor, but you still need a large amount of memory.
To fix this, please download e2fsprogs 1.40.2, and then add to the /etc/e2fsck.conf file (create if it doesn't exist):
[scratch_files]
directory = /var/cache/e2fsck
... and then make sure there is a directory created in /var/cache/e2fsck. This also requires that /var/cache/mounted is mounted read/write by the time the fsck is done, so it won't work for the root filesystem, but normally this isn't an issue for the root filesystem.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks very much for your detailed answer :)
I'm in fact using BackupPC and had around 80 mil Inodes used too.
What I don't understand atm: the memory usage you calculated should be around the same in my case, so 2 GB RAM + 2 GB Swap should be sufficient. Or doesn't e2fsck use swap?
But I may be wrong, I had to swap the Disks and start from scratch to get new Backups running, so I can't check atm. But I will swap the old disks back in next week and have a closer look, and try your solution.
Thanks very much,
Christian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
E2fsck will use swap --- if swap is enabled at the time when it runs. Some distribution boot scripts don't enable swap until later, mainly to accomodate people who prefer to use swap files instead of swap partitions. But even if you are using swap, if you are using a 32-bit x86 system, each process only has 3GB of address space. In addition, things like shared library are generally plopped down in the middle of the address space further breaking up the 3GB of available address space. The issue here is that the 3GB address space has to be shared by the stack, which grows down, and the heap (used for small malloc() allocation) which grows up. So shared libraries, large memory allocations, and mmap() regions are dropped in the middle of the 3GB address space. And if you have large numbers of hard links and directories, this requires large contiguous memory allocations for the arrays to keep track of all the hard links and directories, and that's where you run into trouble, even if you have plenty of physical memory and swap --- on a 32-bit system, you have limited address space.
Another way of solving the problem if this is a new enough machine is to install a x86_64 64-bit system on it, assuming you have enough physical memory --- and this *will* be faster than using [scratch_files]. But if you only have 2GB of physcal memory, I'm guessing this is an older system which isn't 64-bit capable....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
hte-bak2:~# LANG=C fsck.ext3 -vf /dev/sda1
e2fsck 1.40-WIP (14-Nov-2006)
Superblock last mount time is in the future. Fix<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Error allocating directory block array: Memory allocation failed
e2fsck: aborted
Whats the cause for this error, and how can I fix it?
System is on Debian Etch, trying to fix a 1.5 TB Ext3 Partition of a Raid 5 on a 3ware 9550SXU Controller.
2 GB RAM + 2 GB Swap available.
If you need any further information i'll supply it happily, just say so.
Thanks,
Christian
You're probably using something like BackupPC or or pdumpfs which creates a very large number of hard link trees. This exacerbates two memory usage problems:
* Even if directories are equivalent, Unix does not allow directories to
be hardlinked, so if a filesystem has 100,000 directories, each
incremental backup will create 100,000 new directories in the BackupPC
directory. E2fsck requires 12 bytes of storage per directory in order
to store accounting information.
* E2fsck uses an icount abstraction to store the i_links_count
information from the inode, as well as the number of times an inode is
actually referenced by directory. This abstraction uses an
optimization based on the observation that on most normal filesystems,
there are very few hard links (i.e., i_links_count for most regular
files is 1). The icount abstraction uses 6 bytes of memory for each
directory and regular file which has been hardlinked, and two of them
are used.
One such filesystem that was reported to me had 88 million inodes, of which 11 million were files, and 77 million were directories (!). This meant that e2fsck needed to allocate around 881 megabytes of memory in one contiguous array for the dirinfo data structures, and two (approximately) 500 megabyte contiguous arrays for the icount abstraction.
On a 32-bit processor, especially with shared libraries enabled to futher reduce the amount of available 3GB address space, e2fsck can very easily fail to have enough memory. Using a statically-linked e2fsck can help, as can moving to a 64-bit processor, but you still need a large amount of memory.
To fix this, please download e2fsprogs 1.40.2, and then add to the /etc/e2fsck.conf file (create if it doesn't exist):
[scratch_files]
directory = /var/cache/e2fsck
... and then make sure there is a directory created in /var/cache/e2fsck. This also requires that /var/cache/mounted is mounted read/write by the time the fsck is done, so it won't work for the root filesystem, but normally this isn't an issue for the root filesystem.
Thanks very much for your detailed answer :)
I'm in fact using BackupPC and had around 80 mil Inodes used too.
What I don't understand atm: the memory usage you calculated should be around the same in my case, so 2 GB RAM + 2 GB Swap should be sufficient. Or doesn't e2fsck use swap?
But I may be wrong, I had to swap the Disks and start from scratch to get new Backups running, so I can't check atm. But I will swap the old disks back in next week and have a closer look, and try your solution.
Thanks very much,
Christian
E2fsck will use swap --- if swap is enabled at the time when it runs. Some distribution boot scripts don't enable swap until later, mainly to accomodate people who prefer to use swap files instead of swap partitions. But even if you are using swap, if you are using a 32-bit x86 system, each process only has 3GB of address space. In addition, things like shared library are generally plopped down in the middle of the address space further breaking up the 3GB of available address space. The issue here is that the 3GB address space has to be shared by the stack, which grows down, and the heap (used for small malloc() allocation) which grows up. So shared libraries, large memory allocations, and mmap() regions are dropped in the middle of the 3GB address space. And if you have large numbers of hard links and directories, this requires large contiguous memory allocations for the arrays to keep track of all the hard links and directories, and that's where you run into trouble, even if you have plenty of physical memory and swap --- on a 32-bit system, you have limited address space.
Another way of solving the problem if this is a new enough machine is to install a x86_64 64-bit system on it, assuming you have enough physical memory --- and this *will* be faster than using [scratch_files]. But if you only have 2GB of physcal memory, I'm guessing this is an older system which isn't 64-bit capable....