RFI:Emulating large filesystems. - Linux

This is a discussion on RFI:Emulating large filesystems. - Linux ; Hi, I want to write a emulator which can emulate a disk. Basically i want to create large filesystems(~32TB) and large files(~2TB) but i do no have such huge amount of space. I want to know what it takes to ...

+ Reply to Thread
Results 1 to 10 of 10

Thread: RFI:Emulating large filesystems.

  1. RFI:Emulating large filesystems.

    Hi,

    I want to write a emulator which can emulate a disk. Basically i want
    to create large filesystems(~32TB) and large files(~2TB) but i do no
    have such huge amount of space.
    I want to know what it takes to develop such a emulator.
    Can somebody please direct me in this regard.


  2. Re: RFI:Emulating large filesystems.


    santhosh wrote:
    > Hi,
    >
    > I want to write a emulator which can emulate a disk. Basically i want
    > to create large filesystems(~32TB) and large files(~2TB) but i do no
    > have such huge amount of space.
    > I want to know what it takes to develop such a emulator.
    > Can somebody please direct me in this regard.


    Maybe you can look for sparse files. These are mostly empty files,
    which can take up less disk space than their size.


  3. Re: RFI:Emulating large filesystems.

    On a sunny day (17 Sep 2006 03:57:02 -0700) it happened "Sjoerd"
    wrote in
    <1158490622.159657.206600@b28g2000cwb.googlegroups. com>:

    >
    >santhosh wrote:
    >> Hi,
    >>
    >> I want to write a emulator which can emulate a disk. Basically i want
    >> to create large filesystems(~32TB) and large files(~2TB) but i do no
    >> have such huge amount of space.
    >> I want to know what it takes to develop such a emulator.
    >> Can somebody please direct me in this regard.

    >
    >Maybe you can look for sparse files. These are mostly empty files,
    >which can take up less disk space than their size.
    >


    Well, apart from funny things like 'zip it' (hehe),
    why not make an interface with a PC to the internet,
    you could treat sites as files......
    So if 'filesystem' asks for sector 32894987932479732 it does not
    access a /dev/hdd or some other disk, but for example IP xxx.xxx.xxx.xxx
    http page /index.html the first 512 bytes, etc....
    A simple mapping should do.....

    I do not know what the total capacity of the 'net' is atm, but it is bigger
    then the biggest disk I can get.



  4. Re: RFI:Emulating large filesystems.


    Jan Panteltje wrote:
    > On a sunny day (17 Sep 2006 03:57:02 -0700) it happened "Sjoerd"
    > wrote in
    > <1158490622.159657.206600@b28g2000cwb.googlegroups. com>:
    >
    > >
    > >santhosh wrote:
    > >> Hi,
    > >>
    > >> I want to write a emulator which can emulate a disk. Basically i want
    > >> to create large filesystems(~32TB) and large files(~2TB) but i do no
    > >> have such huge amount of space.
    > >> I want to know what it takes to develop such a emulator.
    > >> Can somebody please direct me in this regard.

    > >
    > >Maybe you can look for sparse files. These are mostly empty files,
    > >which can take up less disk space than their size.
    > >

    >
    > Well, apart from funny things like 'zip it' (hehe),
    > why not make an interface with a PC to the internet,
    > you could treat sites as files......
    > So if 'filesystem' asks for sector 32894987932479732 it does not
    > access a /dev/hdd or some other disk, but for example IP xxx.xxx.xxx.xxx
    > http page /index.html the first 512 bytes, etc....
    > A simple mapping should do.....
    >
    > I do not know what the total capacity of the 'net' is atm, but it is bigger
    > then the biggest disk I can get.


    Sparse files does not get any block allocated at all. So if i want to
    seek to say 1TBth byteand check the integrity of the data, it wont be
    possible. Thats why the idea of emulating the disk blocks.


  5. Re: RFI:Emulating large filesystems.

    On a sunny day (17 Sep 2006 04:36:59 -0700) it happened "santhosh"
    wrote in
    <1158493019.510842.106840@i42g2000cwa.googlegroups. com>:

    >
    >Jan Panteltje wrote:
    >> On a sunny day (17 Sep 2006 03:57:02 -0700) it happened "Sjoerd"
    >> wrote in
    >> <1158490622.159657.206600@b28g2000cwb.googlegroups. com>:
    >>
    >> >
    >> >santhosh wrote:
    >> >> Hi,
    >> >>
    >> >> I want to write a emulator which can emulate a disk. Basically i want
    >> >> to create large filesystems(~32TB) and large files(~2TB) but i do no
    >> >> have such huge amount of space.
    >> >> I want to know what it takes to develop such a emulator.
    >> >> Can somebody please direct me in this regard.
    >> >
    >> >Maybe you can look for sparse files. These are mostly empty files,
    >> >which can take up less disk space than their size.
    >> >

    >>
    >> Well, apart from funny things like 'zip it' (hehe),
    >> why not make an interface with a PC to the internet,
    >> you could treat sites as files......
    >> So if 'filesystem' asks for sector 32894987932479732 it does not
    >> access a /dev/hdd or some other disk, but for example IP xxx.xxx.xxx.xxx
    >> http page /index.html the first 512 bytes, etc....
    >> A simple mapping should do.....
    >>
    >> I do not know what the total capacity of the 'net' is atm, but it is bigger
    >> then the biggest disk I can get.

    >
    >Sparse files does not get any block allocated at all. So if i want to
    >seek to say 1TBth byteand check the integrity of the data, it wont be
    >possible. Thats why the idea of emulating the disk blocks.


    Well, I do not know exactly what 'sparse files' are, but if they refer to the
    same data [repeatedly] then how will you even know your sector mapping
    is correct.
    The OP says he want to emulate a _disk_ so then you have only sector numbers
    containing data to play with.
    I do not even see where 'files' come into play.



  6. Re: RFI:Emulating large filesystems.

    On 2006-09-17, Jan Panteltje wrote:

    >>Sparse files does not get any block allocated at all. So if i want to
    >>seek to say 1TBth byteand check the integrity of the data, it wont be
    >>possible. Thats why the idea of emulating the disk blocks.

    >
    > Well, I do not know exactly what 'sparse files' are, but if they refer to the
    > same data [repeatedly] then how will you even know your sector mapping
    > is correct.


    They don't.

    > The OP says he want to emulate a _disk_ so then you have only sector numbers
    > containing data to play with.
    > I do not even see where 'files' come into play.


    What you do is open a file handle seek forwards 32 terrabytes write a single
    sector close the file handle.

    you'll end up with a file that appears to be full of zeroes but those zeroes
    aren't recorded anywhere, rather the most of the file is marked as being
    empty. if you start witing data into the file the parts you write to will
    no longer be empty (and will consume disk space) but remainder of the file will
    only be represented by inodes that say "this block is empty".

    Bye.
    Jasen

  7. Re: RFI:Emulating large filesystems.

    On a sunny day (18 Sep 2006 06:17:44 GMT) it happened jasen
    wrote in :

    >On 2006-09-17, Jan Panteltje wrote:
    >
    >>>Sparse files does not get any block allocated at all. So if i want to
    >>>seek to say 1TBth byteand check the integrity of the data, it wont be
    >>>possible. Thats why the idea of emulating the disk blocks.

    >>
    >> Well, I do not know exactly what 'sparse files' are, but if they refer to the
    >> same data [repeatedly] then how will you even know your sector mapping
    >> is correct.

    >
    >They don't.
    >
    >> The OP says he want to emulate a _disk_ so then you have only sector numbers
    >> containing data to play with.
    >> I do not even see where 'files' come into play.

    >
    >What you do is open a file handle seek forwards 32 terrabytes write a single
    >sector close the file handle.
    >
    >you'll end up with a file that appears to be full of zeroes but those zeroes
    >aren't recorded anywhere, rather the most of the file is marked as being
    >empty. if you start witing data into the file the parts you write to will
    >no longer be empty (and will consume disk space) but remainder of the file will
    >only be represented by inodes that say "this block is empty".
    >
    >Bye.
    > Jasen


    OK, that system I know about, but does not solve the basic problem I think.
    If you write to sector X you create data, if you wanted to test a read from
    all sectors you would still need that much disk space...
    I had an other idea, but did not work it out:
    Say you created an algorithm that makes a 512 bytes of unique data with the
    sector number as 'seed'.
    Problem here is that there is only a limited number of bits, and so
    possibilities, in 512 bytes (sectors), so 8 x 512 bits = 4096 bits.
    So your sector addressing number should be smaller thne a 4096 bit number
    to always have unique data.
    That should be OK with the 1 Terabyte the OP wanted, hey solved this?



  8. Re: RFI:Emulating large filesystems.


    jasen wrote:
    > On 2006-09-17, Jan Panteltje wrote:
    >
    > >>Sparse files does not get any block allocated at all. So if i want to
    > >>seek to say 1TBth byteand check the integrity of the data, it wont be
    > >>possible. Thats why the idea of emulating the disk blocks.

    > >
    > > Well, I do not know exactly what 'sparse files' are, but if they refer to the
    > > same data [repeatedly] then how will you even know your sector mapping
    > > is correct.

    >
    > They don't.
    >
    > > The OP says he want to emulate a _disk_ so then you have only sector numbers
    > > containing data to play with.
    > > I do not even see where 'files' come into play.

    >
    > What you do is open a file handle seek forwards 32 terrabytes write a single
    > sector close the file handle.
    >
    > you'll end up with a file that appears to be full of zeroes but those zeroes
    > aren't recorded anywhere, rather the most of the file is marked as being
    > empty. if you start witing data into the file the parts you write to will
    > no longer be empty (and will consume disk space) but remainder of the file will
    > only be represented by inodes that say "this block is empty".
    >
    > Bye.
    > Jasen


    Hi Jasen,
    In that case, the amount it will still be a partially sparse file.
    du wont report the file as 32TB file, since all the blocks in the file
    are not filled up.
    My basic question is :
    Is there any way in which we can create a largefile(~2TB) without
    having actual disk of 2TB space.


  9. Re: RFI:Emulating large filesystems.

    santhosh wrote:
    > jasen wrote:
    >


    >>What you do is open a file handle seek forwards 32 terrabytes write a single
    >>sector close the file handle.
    >>
    >>you'll end up with a file that appears to be full of zeroes but those zeroes
    >>aren't recorded anywhere, rather the most of the file is marked as being
    >>empty. if you start witing data into the file the parts you write to will
    >>no longer be empty (and will consume disk space) but remainder of the file will
    >>only be represented by inodes that say "this block is empty".
    >>
    >>Bye.
    >> Jasen

    >
    >
    > Hi Jasen,
    > In that case, the amount it will still be a partially sparse file.
    > du wont report the file as 32TB file, since all the blocks in the file
    > are not filled up.
    > My basic question is :
    > Is there any way in which we can create a largefile(~2TB) without
    > having actual disk of 2TB space.


    You mean other than with a sparse file? You mean actually having 2TB of
    _real_ data and a disk with less than 2TB?

    No.

    I mean that are numerous ways to compress data such that it would fit
    into less space, but in general, it is not possible to store more data
    on a disk than will fit.

    NB I take it that by writing "without having actual disk of 2TB space"
    to mean that the disk is _less_ than 2TB (a 4TB disk might qualify for
    not being "actual disk of 2TB space")

    If you leave out the actual data, you could e.g. emulate an insanely
    huge disk by using an existing disk as a bitmap of sectors actually
    written, discarding the data. But then ... you could even do away with
    the disk and pretend that you *want* to write only sectors that have a
    specific structure (which is directly related to the sector's number)
    and you could be able to re-create this structure just by giving the
    sector number. E.g. each sector _always_ holds its sector number and no
    other usefull information.
    --
    Josef Möllers (Pinguinpfleger bei FSC)
    If failure had no penalty success would not be a prize
    -- T. Pratchett


  10. Re: RFI:Emulating large filesystems.

    On 2006-09-19, santhosh wrote:

    >> you'll end up with a file that appears to be full of zeroes but those zeroes
    >> aren't recorded anywhere, rather the most of the file is marked as being
    >> empty. if you start witing data into the file the parts you write to will
    >> no longer be empty (and will consume disk space) but remainder of the file will
    >> only be represented by inodes that say "this block is empty".


    > In that case, the amount it will still be a partially sparse file.
    > du wont report the file as 32TB file, since all the blocks in the file
    > are not filled up.
    > My basic question is :
    > Is there any way in which we can create a largefile(~2TB) without
    > having actual disk of 2TB space.


    create a large sparse file and format it as a partition image then edit the
    inodes and free block bit-map to be consistant with a large file existing on
    it. then mount it.



    Bye.
    Jasen





+ Reply to Thread