Generally speaking we have deduplication (at least but not only) for NTFS and btrfs. Snapraid as a normal user-space app just sees the files as normal files and works fine, except for using more parity space than really "needed". Sure, normally you should use parity drives that are a bit bigger than your bigger disk but if your deduplication is any good you might end up needing MUCH larger disks.
It's clear that's a major undertaking but some way to have this duplicated data take space only once in the parity would help a lot long-term. It might indirectly even help many users that DON'T use any deduplication (or even deduplication capable filesystem) at all! That is in case they have some duplicates those will take less parity so they're less likely to "run out parity" once the largest disk(s) are filled.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Generally speaking we have deduplication (at least but not only) for NTFS and btrfs. Snapraid as a normal user-space app just sees the files as normal files and works fine, except for using more parity space than really "needed". Sure, normally you should use parity drives that are a bit bigger than your bigger disk but if your deduplication is any good you might end up needing MUCH larger disks.
It's clear that's a major undertaking but some way to have this duplicated data take space only once in the parity would help a lot long-term. It might indirectly even help many users that DON'T use any deduplication (or even deduplication capable filesystem) at all! That is in case they have some duplicates those will take less parity so they're less likely to "run out parity" once the largest disk(s) are filled.