This is a discussion on Decentralized/distributed filesystems questions - Storage ; Hi all, I am looking for a decentralized filesystem solution for non-enterprise use (lightweight) to replace several rsync-mirrored, software-based RAID-5 storage servers. I don't know if there's any actual solution out there which meets my fantasy, but here is some ...
I am looking for a decentralized filesystem solution for non-enterprise
use (lightweight) to replace several rsync-mirrored, software-based
RAID-5 storage servers. I don't know if there's any actual solution out
there which meets my fantasy, but here is some criteria that would be
- Linux compatible (obviously)
- Virtual filesystem OK, but real filesystem driver is probably (?) ideal.
- Completely automatic synchronization of data, with automatic
redundancy (like RAID), and the ability to set LEVELS of redundancy for
either the FS as a whole, or certain groups/subsets of files within it.
- Simple/automatic configuration: option to automatically allocate
storage devices, or some % of new storage devices added to a node.
überideal is hardware RAID style: pop in a drive, reboot the machine,
everything is taken care of.
- On a local level, nodes should NOT require a centralized tracker, but
rather find each other via something akin to ZeroConf/Bonjour. Once
found, a node should examine its own tables and, if known, attempt to
authenticate automatically without fuss. On a local level there should
be dynamic IP tolerance (an identifying hash should be used for
- Support for scaling across the Internet would be great.
- In which case, I want to be able to tweak the total "sync" bandwidth
used on each node as can similarly be done with rsync
- And the WHOLE filesystem should be able to be accessed from any node
which allows external connections, anywhere in the world. Chunks of
data which are not local would be transferred via the internet, to that
node, upon demand.
- NO SPECIAL CLIENT SOFTWARE SHOULD BE REQUIRED for accessing shares
(virtual filesystems presumably require a translator installed on the
node in order to provide a mount point?)
This seems to be a combination of a lot of methodologies, but I'm
obviously taking a lot of inspiration from P2P filesharing protocols
such as eDonkey and Gnutella...
Any thoughts? Anything even close to this out there?
Thanks in advance for any help -- should be a good conversation