I think about implementing it using a whole /dev/hdb device (or the 250 GB file within the partition of the device), create the redundancy information in it treating it like an empty one. Subtract the 20%/25%/30%/33% needed for error corrention data and offer it as a virtual rw device.
I don't know which might be easier to implement, Linux or Windows, I would LOVE it on both of course.
With Linux maybe even having root on such a thing might be possible (once the kernel and module is in memory from /boot).
It would lower my worries when my HDD's die slowly, getting early warnings when the correction was needed to read data (popup / evenlog /var/messages /var/syslog etc etc) but still have fully correctable data.
I personally would misuse it for having the need to mirror the device within the machine. I know it does not save me from backup, but it would offer me a more current version of my data when something goes wrong. Would save me from wasting space and money for mirroring on my private servers. It would also make my servers more silent :).
Performance falls under "don't care much" category, today's "small" machines should be fast enough to spit 10 MB/s out of such a thing. They even unpack .zip files in higher speeds than 10 MB/s.
In case something like this already exists , please tell me!
Even more fun (or a distant dream): mixing truecrypt & dvdisaster in one product :).
Setsuna
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think you would do that in RS02 mode, since that would be the only mode to make sense to do such a thing with on a single disk (considering that you're proposing to put the ECC data on the same device or virtual container as the data). But I see a few problems with doing it.
The first problem might be the performance of such a setup. Each time you write to a sector, you would have to re-calculate all the ECC blocks that occur on that sector. As you probably know, an ECC block in dvdisaster contains bytes that are maximally spread out across the container. Updating an ECC block would mean reading back all the data sectors (involving many seeks), calculating the RS sectors, and then writing the RS sectors back out. You'd have at least 255 disk seeks each time you write to a sector. Unless you had some sort of write buffer to go with it, the disk-write throughput would be terrible.
A second problem would be with consistency. What happens to an ECC block if a write operation is interrupted mid-way? If you lose power, or a cable becomes unplugged, or the system crashes. You might end up with inconsistent ECC data. The ECC block may only written half-way, leaving some of the old data present in the ECC block along with some new data. In dvdisaster, the validity of ECC blocks is checked by recording CRCs of every sector. But on a HD, the CRC sectors would themselves have the same problems with consistency just as the ECC blocks do.
When I say "inconsistent", I don't mean just "bad" like you would normally recover from in dvdisaster, I mean you have part of an old ECC block combined with part of a new block, and the ECC data would then be completely useless. It would be important to be able to detect such an inconsistency, by doing something external to the ECC data, like maintaining some kind of ECC journal. If you could not detect inconsistencies and try to recover a block in such a scenario, some data may become ruined.
Some kind of journaling system would have to be created to avoid problems with consistency (this has nothing to do with the journaling that occurs within the filesystem itself inside the container).
Considering all that, it'd probably still be much more effective to implement a RAID-5 instead. You'd have your redundancy and recoverability, but without the performance and consistency issues you'd have by keeping up with RS02 data.
As far as putting truecrypt & dvdisaster together, you can sorta do that now, although read-only. By creating a raw truecrypt container just slightly larger than your dvdisaster image (512 bytes larger), and dd'ing the dvdisaster image into it. But I think you mean you'd like to have full RW functionality while doing that, which you couldn't do (truecrypt just came out with DVDRW support, but you still have the issue of generating the dvdisaster image, which means you can't RW). Then to get the data back out, you could just mount the truecrypt container, and dd out the dvdisaster image, or mount it directly.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Since it works so nicely on .iso images..
I think about implementing it using a whole /dev/hdb device (or the 250 GB file within the partition of the device), create the redundancy information in it treating it like an empty one. Subtract the 20%/25%/30%/33% needed for error corrention data and offer it as a virtual rw device.
I don't know which might be easier to implement, Linux or Windows, I would LOVE it on both of course.
With Linux maybe even having root on such a thing might be possible (once the kernel and module is in memory from /boot).
It would lower my worries when my HDD's die slowly, getting early warnings when the correction was needed to read data (popup / evenlog /var/messages /var/syslog etc etc) but still have fully correctable data.
I personally would misuse it for having the need to mirror the device within the machine. I know it does not save me from backup, but it would offer me a more current version of my data when something goes wrong. Would save me from wasting space and money for mirroring on my private servers. It would also make my servers more silent :).
Performance falls under "don't care much" category, today's "small" machines should be fast enough to spit 10 MB/s out of such a thing. They even unpack .zip files in higher speeds than 10 MB/s.
In case something like this already exists , please tell me!
Even more fun (or a distant dream): mixing truecrypt & dvdisaster in one product :).
Setsuna
I think you would do that in RS02 mode, since that would be the only mode to make sense to do such a thing with on a single disk (considering that you're proposing to put the ECC data on the same device or virtual container as the data). But I see a few problems with doing it.
The first problem might be the performance of such a setup. Each time you write to a sector, you would have to re-calculate all the ECC blocks that occur on that sector. As you probably know, an ECC block in dvdisaster contains bytes that are maximally spread out across the container. Updating an ECC block would mean reading back all the data sectors (involving many seeks), calculating the RS sectors, and then writing the RS sectors back out. You'd have at least 255 disk seeks each time you write to a sector. Unless you had some sort of write buffer to go with it, the disk-write throughput would be terrible.
A second problem would be with consistency. What happens to an ECC block if a write operation is interrupted mid-way? If you lose power, or a cable becomes unplugged, or the system crashes. You might end up with inconsistent ECC data. The ECC block may only written half-way, leaving some of the old data present in the ECC block along with some new data. In dvdisaster, the validity of ECC blocks is checked by recording CRCs of every sector. But on a HD, the CRC sectors would themselves have the same problems with consistency just as the ECC blocks do.
When I say "inconsistent", I don't mean just "bad" like you would normally recover from in dvdisaster, I mean you have part of an old ECC block combined with part of a new block, and the ECC data would then be completely useless. It would be important to be able to detect such an inconsistency, by doing something external to the ECC data, like maintaining some kind of ECC journal. If you could not detect inconsistencies and try to recover a block in such a scenario, some data may become ruined.
Some kind of journaling system would have to be created to avoid problems with consistency (this has nothing to do with the journaling that occurs within the filesystem itself inside the container).
Considering all that, it'd probably still be much more effective to implement a RAID-5 instead. You'd have your redundancy and recoverability, but without the performance and consistency issues you'd have by keeping up with RS02 data.
As far as putting truecrypt & dvdisaster together, you can sorta do that now, although read-only. By creating a raw truecrypt container just slightly larger than your dvdisaster image (512 bytes larger), and dd'ing the dvdisaster image into it. But I think you mean you'd like to have full RW functionality while doing that, which you couldn't do (truecrypt just came out with DVDRW support, but you still have the issue of generating the dvdisaster image, which means you can't RW). Then to get the data back out, you could just mount the truecrypt container, and dd out the dvdisaster image, or mount it directly.