Printable View
Hi,
I want to write a emulator which can emulate a disk. Basically i want
to create large filesystems(~32TB) and large files(~2TB) but i do no
have such huge amount of space.
I want to know what it takes to develop such a emulator.
Can somebody please direct me in this regard.
santhosh wrote:[color=blue]
> Hi,
>
> I want to write a emulator which can emulate a disk. Basically i want
> to create large filesystems(~32TB) and large files(~2TB) but i do no
> have such huge amount of space.
> I want to know what it takes to develop such a emulator.
> Can somebody please direct me in this regard.[/color]
Maybe you can look for sparse files. These are mostly empty files,
which can take up less disk space than their size.
On a sunny day (17 Sep 2006 03:57:02 -0700) it happened "Sjoerd"
<sjoerder@gmail.com> wrote in
<1158490622.159657.206600@b28g2000cwb.googlegroups.com>:
[color=blue]
>
>santhosh wrote:[color=green]
>> Hi,
>>
>> I want to write a emulator which can emulate a disk. Basically i want
>> to create large filesystems(~32TB) and large files(~2TB) but i do no
>> have such huge amount of space.
>> I want to know what it takes to develop such a emulator.
>> Can somebody please direct me in this regard.[/color]
>
>Maybe you can look for sparse files. These are mostly empty files,
>which can take up less disk space than their size.
>[/color]
Well, apart from funny things like 'zip it' (hehe),
why not make an interface with a PC to the internet,
you could treat sites as files......
So if 'filesystem' asks for sector 32894987932479732 it does not
access a /dev/hdd or some other disk, but for example IP xxx.xxx.xxx.xxx
http page /index.html the first 512 bytes, etc....
A simple mapping should do.....
I do not know what the total capacity of the 'net' is atm, but it is bigger
then the biggest disk I can get.
Jan Panteltje wrote:[color=blue]
> On a sunny day (17 Sep 2006 03:57:02 -0700) it happened "Sjoerd"
> <sjoerder@gmail.com> wrote in
> <1158490622.159657.206600@b28g2000cwb.googlegroups.com>:
>[color=green]
> >
> >santhosh wrote:[color=darkred]
> >> Hi,
> >>
> >> I want to write a emulator which can emulate a disk. Basically i want
> >> to create large filesystems(~32TB) and large files(~2TB) but i do no
> >> have such huge amount of space.
> >> I want to know what it takes to develop such a emulator.
> >> Can somebody please direct me in this regard.[/color]
> >
> >Maybe you can look for sparse files. These are mostly empty files,
> >which can take up less disk space than their size.
> >[/color]
>
> Well, apart from funny things like 'zip it' (hehe),
> why not make an interface with a PC to the internet,
> you could treat sites as files......
> So if 'filesystem' asks for sector 32894987932479732 it does not
> access a /dev/hdd or some other disk, but for example IP xxx.xxx.xxx.xxx
> http page /index.html the first 512 bytes, etc....
> A simple mapping should do.....
>
> I do not know what the total capacity of the 'net' is atm, but it is bigger
> then the biggest disk I can get.[/color]
Sparse files does not get any block allocated at all. So if i want to
seek to say 1TBth byteand check the integrity of the data, it wont be
possible. Thats why the idea of emulating the disk blocks.
On a sunny day (17 Sep 2006 04:36:59 -0700) it happened "santhosh"
<santhoshkumarpa@gmail.com> wrote in
<1158493019.510842.106840@i42g2000cwa.googlegroups.com>:
[color=blue]
>
>Jan Panteltje wrote:[color=green]
>> On a sunny day (17 Sep 2006 03:57:02 -0700) it happened "Sjoerd"
>> <sjoerder@gmail.com> wrote in
>> <1158490622.159657.206600@b28g2000cwb.googlegroups.com>:
>>[color=darkred]
>> >
>> >santhosh wrote:
>> >> Hi,
>> >>
>> >> I want to write a emulator which can emulate a disk. Basically i want
>> >> to create large filesystems(~32TB) and large files(~2TB) but i do no
>> >> have such huge amount of space.
>> >> I want to know what it takes to develop such a emulator.
>> >> Can somebody please direct me in this regard.
>> >
>> >Maybe you can look for sparse files. These are mostly empty files,
>> >which can take up less disk space than their size.
>> >[/color]
>>
>> Well, apart from funny things like 'zip it' (hehe),
>> why not make an interface with a PC to the internet,
>> you could treat sites as files......
>> So if 'filesystem' asks for sector 32894987932479732 it does not
>> access a /dev/hdd or some other disk, but for example IP xxx.xxx.xxx.xxx
>> http page /index.html the first 512 bytes, etc....
>> A simple mapping should do.....
>>
>> I do not know what the total capacity of the 'net' is atm, but it is bigger
>> then the biggest disk I can get.[/color]
>
>Sparse files does not get any block allocated at all. So if i want to
>seek to say 1TBth byteand check the integrity of the data, it wont be
>possible. Thats why the idea of emulating the disk blocks.[/color]
Well, I do not know exactly what 'sparse files' are, but if they refer to the
same data [repeatedly] then how will you even know your sector mapping
is correct.
The OP says he want to emulate a _disk_ so then you have only sector numbers
containing data to play with.
I do not even see where 'files' come into play.
On 2006-09-17, Jan Panteltje <pNaonStpealmtje@yahoo.com> wrote:
[color=blue][color=green]
>>Sparse files does not get any block allocated at all. So if i want to
>>seek to say 1TBth byteand check the integrity of the data, it wont be
>>possible. Thats why the idea of emulating the disk blocks.[/color]
>
> Well, I do not know exactly what 'sparse files' are, but if they refer to the
> same data [repeatedly] then how will you even know your sector mapping
> is correct.[/color]
They don't.
[color=blue]
> The OP says he want to emulate a _disk_ so then you have only sector numbers
> containing data to play with.
> I do not even see where 'files' come into play.[/color]
What you do is open a file handle seek forwards 32 terrabytes write a single
sector close the file handle.
you'll end up with a file that appears to be full of zeroes but those zeroes
aren't recorded anywhere, rather the most of the file is marked as being
empty. if you start witing data into the file the parts you write to will
no longer be empty (and will consume disk space) but remainder of the file will
only be represented by inodes that say "this block is empty".
Bye.
Jasen
On a sunny day (18 Sep 2006 06:17:44 GMT) it happened jasen
<jasen@free.net.nz> wrote in <eeldm8$gon$1@gonzo.homenet>:
[color=blue]
>On 2006-09-17, Jan Panteltje <pNaonStpealmtje@yahoo.com> wrote:
>[color=green][color=darkred]
>>>Sparse files does not get any block allocated at all. So if i want to
>>>seek to say 1TBth byteand check the integrity of the data, it wont be
>>>possible. Thats why the idea of emulating the disk blocks.[/color]
>>
>> Well, I do not know exactly what 'sparse files' are, but if they refer to the
>> same data [repeatedly] then how will you even know your sector mapping
>> is correct.[/color]
>
>They don't.
>[color=green]
>> The OP says he want to emulate a _disk_ so then you have only sector numbers
>> containing data to play with.
>> I do not even see where 'files' come into play.[/color]
>
>What you do is open a file handle seek forwards 32 terrabytes write a single
>sector close the file handle.
>
>you'll end up with a file that appears to be full of zeroes but those zeroes
>aren't recorded anywhere, rather the most of the file is marked as being
>empty. if you start witing data into the file the parts you write to will
>no longer be empty (and will consume disk space) but remainder of the file will
>only be represented by inodes that say "this block is empty".
>
>Bye.
> Jasen[/color]
OK, that system I know about, but does not solve the basic problem I think.
If you write to sector X you create data, if you wanted to test a read from
all sectors you would still need that much disk space...
I had an other idea, but did not work it out:
Say you created an algorithm that makes a 512 bytes of unique data with the
sector number as 'seed'.
Problem here is that there is only a limited number of bits, and so
possibilities, in 512 bytes (sectors), so 8 x 512 bits = 4096 bits.
So your sector addressing number should be smaller thne a 4096 bit number
to always have unique data.
That should be OK with the 1 Terabyte the OP wanted, hey solved this?
jasen wrote:[color=blue]
> On 2006-09-17, Jan Panteltje <pNaonStpealmtje@yahoo.com> wrote:
>[color=green][color=darkred]
> >>Sparse files does not get any block allocated at all. So if i want to
> >>seek to say 1TBth byteand check the integrity of the data, it wont be
> >>possible. Thats why the idea of emulating the disk blocks.[/color]
> >
> > Well, I do not know exactly what 'sparse files' are, but if they refer to the
> > same data [repeatedly] then how will you even know your sector mapping
> > is correct.[/color]
>
> They don't.
>[color=green]
> > The OP says he want to emulate a _disk_ so then you have only sector numbers
> > containing data to play with.
> > I do not even see where 'files' come into play.[/color]
>
> What you do is open a file handle seek forwards 32 terrabytes write a single
> sector close the file handle.
>
> you'll end up with a file that appears to be full of zeroes but those zeroes
> aren't recorded anywhere, rather the most of the file is marked as being
> empty. if you start witing data into the file the parts you write to will
> no longer be empty (and will consume disk space) but remainder of the file will
> only be represented by inodes that say "this block is empty".
>
> Bye.
> Jasen[/color]
Hi Jasen,
In that case, the amount it will still be a partially sparse file.
du wont report the file as 32TB file, since all the blocks in the file
are not filled up.
My basic question is :
Is there any way in which we can create a largefile(~2TB) without
having actual disk of 2TB space.
santhosh wrote:[color=blue]
> jasen wrote:
> [/color]
[color=blue][color=green]
>>What you do is open a file handle seek forwards 32 terrabytes write a single
>>sector close the file handle.
>>
>>you'll end up with a file that appears to be full of zeroes but those zeroes
>>aren't recorded anywhere, rather the most of the file is marked as being
>>empty. if you start witing data into the file the parts you write to will
>>no longer be empty (and will consume disk space) but remainder of the file will
>>only be represented by inodes that say "this block is empty".
>>
>>Bye.
>> Jasen[/color]
>
>
> Hi Jasen,
> In that case, the amount it will still be a partially sparse file.
> du wont report the file as 32TB file, since all the blocks in the file
> are not filled up.
> My basic question is :
> Is there any way in which we can create a largefile(~2TB) without
> having actual disk of 2TB space.[/color]
You mean other than with a sparse file? You mean actually having 2TB of
_real_ data and a disk with less than 2TB?
No.
I mean that are numerous ways to compress data such that it would fit
into less space, but in general, it is not possible to store more data
on a disk than will fit.
NB I take it that by writing "without having actual disk of 2TB space"
to mean that the disk is _less_ than 2TB (a 4TB disk might qualify for
not being "actual disk of 2TB space")
If you leave out the actual data, you could e.g. emulate an insanely
huge disk by using an existing disk as a bitmap of sectors actually
written, discarding the data. But then ... you could even do away with
the disk and pretend that you *want* to write only sectors that have a
specific structure (which is directly related to the sector's number)
and you could be able to re-create this structure just by giving the
sector number. E.g. each sector _always_ holds its sector number and no
other usefull information.
--
Josef Möllers (Pinguinpfleger bei FSC)
If failure had no penalty success would not be a prize
-- T. Pratchett
On 2006-09-19, santhosh <santhoshkumarpa@gmail.com> wrote:
[color=blue][color=green]
>> you'll end up with a file that appears to be full of zeroes but those zeroes
>> aren't recorded anywhere, rather the most of the file is marked as being
>> empty. if you start witing data into the file the parts you write to will
>> no longer be empty (and will consume disk space) but remainder of the file will
>> only be represented by inodes that say "this block is empty".[/color][/color]
[color=blue]
> In that case, the amount it will still be a partially sparse file.
> du wont report the file as 32TB file, since all the blocks in the file
> are not filled up.
> My basic question is :
> Is there any way in which we can create a largefile(~2TB) without
> having actual disk of 2TB space.[/color]
create a large sparse file and format it as a partition image then edit the
inodes and free block bit-map to be consistant with a large file existing on
it. then mount it.
Bye.
Jasen