I have found an error in the way that LiveCD install to Hard drive works.
I got NST and, being pleased with how well it worked from the Live CD, obtained a 16 gig USB stick, which I would install onto.
The plan was to install NST onto about 11 gig of the pen drive (300 MB /boot, 9.5 gig or so /, 1 gig swap). However, no matter what I did, it would not install. Anaconda would continually complain that there was an error writing to the disk.
I tried everything I could think of, investigated possible bugs in anaconda, just couldn't understand it.
If I mounted the resulting filesystem, it was 100% full, but way way smaller than the 9.5 gigabytes. It was about the size of the occupied space on the hard drive.
I tried installing using defaults - still wouldn't work!
Finally, this morning, after another two failed installs, I figured out the problem. I'm still not sure where the problem lies, and suspect it's in the way the LiveCD has loopbacks set up, but I have found it …
On boot, if you issue the 'df' command, you'll see (on my systems anyway - 4 gig ram) that / is mounted on /dev/mapper/live-rw and reports a size of about 5.2 gigabytes with 5 gig or so free.
The anaconda process is supposed to copy the read-only /dev/mapper/live-osimg-min onto the USB stick; this was verified by going and looking at what livecd.py does, and doing other research into the install process. Looking at the install process, this is indeed what it seems to be doing. But, something strange is happening.
If you manually use 'dd' to copy the live-osimg-min image onto the 9.5 gig partition (where it should fit comfortably), you will find that, as well as it taking a very very long time, it will eventually fall over with 'No space left on device' after filling up the partition. It is copying over not only the read only image, but also all the filesystem free space for some reason!
This explains why the installer gets to 99% and then sits there for … about as long as it ran in the first place - because it's copying over an amount of free space almost equal to the size of live-osimg-min.
Steps to reproduce:
- Obtain a 16 gig USB stick
- run the installer
- Accept defaults for partitioning, then look at file system layout. On mine, the defaults ended up at 500 MB boot, 9.something for /, and approximately 5.3 Gb for swap (Exactly why it thinks I need so much swap is a mystery). If yours don't quite come out at that, just go try and make / 9 Gig or so (although, make it, say, 6.5 for testing - that way it will fail quicker).
- Watch the install process fail.
- verify that this is a mistake by mounting /dev/XXXX and doing a 'df' on it; you should see the size of the filesystem does not include the free space.
- further verify that livecd-osimg-min is also taking over free space by running 'dd if=/dev/mapper/livecd-os-min of=/dev/XXXX bs=8M and watch it fall over long after it should have finished copying, reporting a number of bytes copied which is larger than the filesystem.
Workaround: Make sure the / partition is slightly greater than the FULL SIZE of rootfs (including its free space). Especially, check that the system has not allocated huge amounts to swap.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No, it doesn't, because either it contains numerous inaccuracies, or the ext3fs.img is the root cause of the wiki and reality being inconsistent. You will note from 'ls' above that the extfs3.img is less than a week old. I did refer to that wiki page when attempting to install.
So … according to the wiki:
What is the minimum size memory stick for a USB Full Install?
8 GB.
Wrong. The system requires / to be at least approximately ten gigabytes, because it is replicating the empty space in extfs3.img as well as the actual used space.
My original object was to have NST take up approximately 10 gig of the USB memory stick (which would be plenty enough) and leave enough FAT32 formatted space on it (for general transfer of stuff between systems) to hold a DVD iso image.
Instead, I have had a whole weekend of frustration, including staying up till 1:30 am, trying to get this to work.
Please, examine the size of ext3fs.img within squashfs. It clearly has too much free space in the filesystem.
livecd.py does not believe that the filesystem should contain any free space (this will be why it hangs at 99% for almost as long as it took to get there - it's replicating the free space).
Set the partition size for / to be about 9 Gb and try and install. You'll soon hit the same problem I did.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK … so I see I have a very new distro, you only released a brand new version last week … there are bound to be some teething troubles I guess. Trust me to be the first to find out :/
I would update the wiki so that it's now accurate, but I can't. If you would be so kind as to update the wiki to reflect the new space requirements for / partition, that would be wonderful. However, at least now that I've posted up about this, there's something for others to see if anyone else tries what I did and fails …
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm sorry to hear that you ran across this issue. You are correct in that the minimum size stated on the Wiki of 8 GB is too small. I have corrected both the USB and and hard disk installation size to state that 10 GB is required.
To be honest, I had not realized that Fedora's Anaconda installer would care about the total allocated size of the disk image on the ISO (which we bumped to 10 GB). My assumption was that it would only care about the 5 GB of actual storage used (plus a bit more).
We had a general rule of thumb to keep our disk images at roughly 50% capacity. I'm not exactly certain why we had this rule of thumb and will discuss it with Ron. We will think about reducing it in the future.
If you are building your ISO from source, you can play with the following setting in the livecd/include/make/iso.mk file (the value is in MB):
PART_SIZE = 10240
Let me know if you would like a account on the NST Wiki site. Due to spam issues we no longer allow people to create accounts on the fly.
Paul
and rebuild an ISO ima
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks. I look forwards to seeing what I can do with it. It's a very nice package - sorry I got a bit ratty about it not working :)
If you look at Anaconda's livecd.py, you'll see it just copies the device in 8 megabyte chunks. Its progress bar goes by the reported size used (since it believes it would only ever be copying a read only file system), which means that during install the progress bar sticks at 99% for a very long time (something that should maybe also be mentioned in the wiki so that people realise it will complete eventually); it's just doing a block read so it has no idea when it's actually finished until the device stops returning blocks …
And I don't know if you could have it cut off when it reaches the reported size used, unless you could guarantee all valid data in ext3fs.img was contigious from the start of the image. And even then, the filesystem would have issues with missing metadata.
Now, the question is: have other livecd-based distros solved this problem in any way?
Also, does this count as a bug in anaconda that should be brought to the attention of the Anaconda maintainers, or would they just say 'we don't expect the image to have free space'? I don't know enough Python (I know very little …) to figure out exactly where it determines the space occupied by the image…
Could livecd.py do something cleverer, like init a new fs on the target device and do a cpio pipeline to backup source device and restore to target? How much would that slow down installation, though? Could livecd.py be clever enough to take that strategy in cases like this, where it detects it can't fit on to the target?
Could it shrink the filesystem before install (I tried, but couldn't get that to work online… you'd probably need a special LiveCD install boot mode that shrunk the filesystem before booting, if that's even possible, given it's readonly and coming from an ISO)?
I don't have the Linux based resources at the moment to build an ISO from scratch. It's just one of those learning experiences.
I don't know if I need an account on the wiki myself, but I'll let you know. Thanks again though :)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We are not really interested with messing with the Anaconda source ourselves (we have had to make patches in the past and it gets to be a pain to maintain).
We did decide to drop the initial size down to 7GB so that it should be possible to install it onto a 8GB USB thumb drive again. These changes have been pushed out to the public Subversion repository and are available now for those that build the NST from source.
We will take a look at this again when we move from Fedora 16 to Fedora 18.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I have found an error in the way that LiveCD install to Hard drive works.
I got NST and, being pleased with how well it worked from the Live CD, obtained a 16 gig USB stick, which I would install onto.
The plan was to install NST onto about 11 gig of the pen drive (300 MB /boot, 9.5 gig or so /, 1 gig swap). However, no matter what I did, it would not install. Anaconda would continually complain that there was an error writing to the disk.
I tried everything I could think of, investigated possible bugs in anaconda, just couldn't understand it.
If I mounted the resulting filesystem, it was 100% full, but way way smaller than the 9.5 gigabytes. It was about the size of the occupied space on the hard drive.
I tried installing using defaults - still wouldn't work!
Finally, this morning, after another two failed installs, I figured out the problem. I'm still not sure where the problem lies, and suspect it's in the way the LiveCD has loopbacks set up, but I have found it …
On boot, if you issue the 'df' command, you'll see (on my systems anyway - 4 gig ram) that / is mounted on /dev/mapper/live-rw and reports a size of about 5.2 gigabytes with 5 gig or so free.
The anaconda process is supposed to copy the read-only /dev/mapper/live-osimg-min onto the USB stick; this was verified by going and looking at what livecd.py does, and doing other research into the install process. Looking at the install process, this is indeed what it seems to be doing. But, something strange is happening.
If you manually use 'dd' to copy the live-osimg-min image onto the 9.5 gig partition (where it should fit comfortably), you will find that, as well as it taking a very very long time, it will eventually fall over with 'No space left on device' after filling up the partition. It is copying over not only the read only image, but also all the filesystem free space for some reason!
This explains why the installer gets to 99% and then sits there for … about as long as it ran in the first place - because it's copying over an amount of free space almost equal to the size of live-osimg-min.
Steps to reproduce:
- Obtain a 16 gig USB stick
- run the installer
- Accept defaults for partitioning, then look at file system layout. On mine, the defaults ended up at 500 MB boot, 9.something for /, and approximately 5.3 Gb for swap (Exactly why it thinks I need so much swap is a mystery). If yours don't quite come out at that, just go try and make / 9 Gig or so (although, make it, say, 6.5 for testing - that way it will fail quicker).
- Watch the install process fail.
- verify that this is a mistake by mounting /dev/XXXX and doing a 'df' on it; you should see the size of the filesystem does not include the free space.
- further verify that livecd-osimg-min is also taking over free space by running 'dd if=/dev/mapper/livecd-os-min of=/dev/XXXX bs=8M and watch it fall over long after it should have finished copying, reporting a number of bytes copied which is larger than the filesystem.
Workaround: Make sure the / partition is slightly greater than the FULL SIZE of rootfs (including its free space). Especially, check that the system has not allocated huge amounts to swap.
I have tracked down the problem a bit further.
It's because the ext3fs.img inside the squashfs.img is 10 gig in size with 5 gig of free space. live-osimg-min just appears to be a stub.
I have tracked down the problem a bit further.
It's because the ext3fs.img inside the squashfs.img is 10 gig in size with 5 gig of free space. live-osimg-min just appears to be a stub.
Sorry about the double post above …
# mkdir /mnt/tmp
# mount -o loop /media/nst-2.16.0-4104.i686/LiveOS/squashfs.img /mnt/tmp
# df /mnt/tmp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/loop0 1456640 1456640 0 100% /mnt/tmp
# ls -l /mnt/tmp/LiveOS/
total 5214081
-rwxr-xr-x. 1 root root 10737418240 Sep 3 19:35 ext3fs.img
# mkdir /mnt/tmp2
# mount -o loop /mnt/tmp/LiveOS/ext3fs.img /mnt/tmp2
# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 13835344 5514004 8180788 41% /
/dev/loop0 1456640 1456640 0 100% /mnt/tmp
/dev/loop1 10321208 5170544 5045816 51% /mnt/tmp2
# ls /mnt/tmp
LiveOS
macartm:
Hopefully this NST Wiki page will help you:
http://wiki.networksecuritytoolkit.org/nstwiki/index.php/NST_USB_FAQ
--RWH
Hi,
No, it doesn't, because either it contains numerous inaccuracies, or the ext3fs.img is the root cause of the wiki and reality being inconsistent. You will note from 'ls' above that the extfs3.img is less than a week old. I did refer to that wiki page when attempting to install.
So … according to the wiki:
What is the minimum size memory stick for a USB Full Install?
8 GB.
Wrong. The system requires / to be at least approximately ten gigabytes, because it is replicating the empty space in extfs3.img as well as the actual used space.
My original object was to have NST take up approximately 10 gig of the USB memory stick (which would be plenty enough) and leave enough FAT32 formatted space on it (for general transfer of stuff between systems) to hold a DVD iso image.
Instead, I have had a whole weekend of frustration, including staying up till 1:30 am, trying to get this to work.
Please, examine the size of ext3fs.img within squashfs. It clearly has too much free space in the filesystem.
livecd.py does not believe that the filesystem should contain any free space (this will be why it hangs at 99% for almost as long as it took to get there - it's replicating the free space).
Set the partition size for / to be about 9 Gb and try and install. You'll soon hit the same problem I did.
OK … so I see I have a very new distro, you only released a brand new version last week … there are bound to be some teething troubles I guess. Trust me to be the first to find out :/
I would update the wiki so that it's now accurate, but I can't. If you would be so kind as to update the wiki to reflect the new space requirements for / partition, that would be wonderful. However, at least now that I've posted up about this, there's something for others to see if anyone else tries what I did and fails …
I'm sorry to hear that you ran across this issue. You are correct in that the minimum size stated on the Wiki of 8 GB is too small. I have corrected both the USB and and hard disk installation size to state that 10 GB is required.
To be honest, I had not realized that Fedora's Anaconda installer would care about the total allocated size of the disk image on the ISO (which we bumped to 10 GB). My assumption was that it would only care about the 5 GB of actual storage used (plus a bit more).
We had a general rule of thumb to keep our disk images at roughly 50% capacity. I'm not exactly certain why we had this rule of thumb and will discuss it with Ron. We will think about reducing it in the future.
If you are building your ISO from source, you can play with the following setting in the livecd/include/make/iso.mk file (the value is in MB):
PART_SIZE = 10240
Let me know if you would like a account on the NST Wiki site. Due to spam issues we no longer allow people to create accounts on the fly.
Paul
and rebuild an ISO ima
Thanks. I look forwards to seeing what I can do with it. It's a very nice package - sorry I got a bit ratty about it not working :)
If you look at Anaconda's livecd.py, you'll see it just copies the device in 8 megabyte chunks. Its progress bar goes by the reported size used (since it believes it would only ever be copying a read only file system), which means that during install the progress bar sticks at 99% for a very long time (something that should maybe also be mentioned in the wiki so that people realise it will complete eventually); it's just doing a block read so it has no idea when it's actually finished until the device stops returning blocks …
And I don't know if you could have it cut off when it reaches the reported size used, unless you could guarantee all valid data in ext3fs.img was contigious from the start of the image. And even then, the filesystem would have issues with missing metadata.
Now, the question is: have other livecd-based distros solved this problem in any way?
Also, does this count as a bug in anaconda that should be brought to the attention of the Anaconda maintainers, or would they just say 'we don't expect the image to have free space'? I don't know enough Python (I know very little …) to figure out exactly where it determines the space occupied by the image…
Could livecd.py do something cleverer, like init a new fs on the target device and do a cpio pipeline to backup source device and restore to target? How much would that slow down installation, though? Could livecd.py be clever enough to take that strategy in cases like this, where it detects it can't fit on to the target?
Could it shrink the filesystem before install (I tried, but couldn't get that to work online… you'd probably need a special LiveCD install boot mode that shrunk the filesystem before booting, if that's even possible, given it's readonly and coming from an ISO)?
I don't have the Linux based resources at the moment to build an ISO from scratch. It's just one of those learning experiences.
I don't know if I need an account on the wiki myself, but I'll let you know. Thanks again though :)
Just as a follow up to this note:
We are not really interested with messing with the Anaconda source ourselves (we have had to make patches in the past and it gets to be a pain to maintain).
We did decide to drop the initial size down to 7GB so that it should be possible to install it onto a 8GB USB thumb drive again. These changes have been pushed out to the public Subversion repository and are available now for those that build the NST from source.
We will take a look at this again when we move from Fedora 16 to Fedora 18.