Faking dense files...

To all,

Thank you again for all of the ideas. Perhaps this will give a
clear picture of what I had in mind. This article describes a patch
for ext3 that allows for block reservation.

[url]http://lwn.net/Articles/81357/[/url]

It doesn't look like this made it into the main ext3 code, but
perhaps for ext4... While it sounds from the article like it's not
completely "tight" ("...will not actually prevent blocks inside the
window from being allocated to some other file"), this is the kind of
thing I was originally asking about. In particular, since it appears
to be done in core without any disk I/O, it should be very quick, even
for very large files.

So I'm getting the feeling that what I'm asking about (quick,
in-core block reservation) is certainly not standard or even available
in the form I'm looking for currently. However, I'll keep my eyes
peeled for ext4 and/or other file systems that have this feature, and,
when it shows up, use that as the recommended file system for installs
of my app. While this won't cover all systems, it might be the best I
can do. Thank you all again for the input.

Allan Stirrett

[email]phil-news-nospam@ipal.net[/email] writes:[color=blue]
> In comp.os.linux.embedded Rainer Weikusat <rainer.weikusat@sncag.com> wrote:[/color]

[...]
[color=blue]
> | I understand what he would like to do, but that doesn't change the
> | fact that this is basically a policy-descision made by someone using
> | the respective applications. That someone needs to have enough disk
> | space available for all concurrently running tasks and if there isn't
> | enough disk space available, some or all of them will fail and the
> | solution is to either not have different applications compete for
> | resources that are scarce enough that there will be a 'loser' and a
> | 'winner' or to accept that this will happen.
>
> Competing for resource is not an uncommon issue. But such competition
> does operate more gracefully if the latter tasks can at least know BEFORE
> they take certain steps, that the space is committed to them.[/color]

This statement boils down to 'not competing for ressources is more
graceful than competing for ressources'. But if application A is so
much more important than application B that the latter should always
fail if the former is active, why the heck are they running
concurrently and compete for ressources in the first place?

Judging from the other end: If application A needs several hours to
complete, why should it voluntarily refuse to work because of a
condition that was true at the time someone wanted it to work, but may
no longer be true several hours later?

Hello,

Allan Stirrett wrote:
[color=blue]
> Thank you again for all of the ideas. Perhaps this will give a
> clear picture of what I had in mind. This article describes a patch
> for ext3 that allows for block reservation.
>
> [url]http://lwn.net/Articles/81357/[/url][/color]

Besides that this reservation was not supposed to be for large amounts,
only for relatively small numbers of blocks to avoid fragmentation, and
it was meant to be done without user interaction.
[color=blue]
> So I'm getting the feeling that what I'm asking about (quick,
> in-core block reservation) is certainly not standard or even available
> in the form I'm looking for currently.[/color]

Reserving large amounts of blocks is strictly contrary to the goals of
an OS. This can only fail, because if every app always uses the upper
bound for reservation before any actual operation, resources will seem
exhausted a lot earlier before anything is really exhausted.

If you used more and smaller files instead of one big, you could more
easily spread over more discs avoiding the race. Maybe it could be
arranged more easily that the initial files could be consumed before
the later files fill the space, e.g. you could wait for space to become
available for the later files. If there is a point in time, where you
need more disk space than available, then no trick will help you over
the fact that you need more discs.

[color=blue]
> However, I'll keep my eyes
> peeled for ext4 and/or other file systems that have this feature, and,
> when it shows up, use that as the recommended file system for installs
> of my app. While this won't cover all systems, it might be the best I
> can do. Thank you all again for the input.[/color]

This is no filesystem problem, it is an application problem to handle
resources properly, and an administration problem to provide enough
resources.

Bernd Strieder

On 11 Jan 2007 20:24:05 GMT, [email]phil-news-nospam@ipal.net[/email] wrote:
[color=blue]
>In comp.os.linux.embedded Allan Stirrett <astirret@tandbergtv.com> wrote:
>
>| To that end, we need a method to reliably and portably have the file
>| system reserve those blocks for us when we START. However, I'm getting
>| the impression that currently it can't be done in a timely-enough
>| fashion for files as large as we're working with.
>
>Writing out commit blocks to disk is certainly not instant, but it is
>a lot faster than many networks. Just write some zeros with dd and time
>it to see how long.
>
>Another alternative is to sufficient partition the disk and ensure that
>only a certain program is allowed to write in that space. That program
>then manages the space numerically to reserve space rather than using the
>filesystem to do so. It will need to also know when the space is released
>so it can increment the available number back up.
>
>--[/color]

In an embedded system that must not fail, most people grab all the
resources their app needs at once. If it fails to get them, it would
fail eventually anyway ... better to fail quickly.

So the best solution I have seen here is to partition your disk. Make
one partition just for your data collection app and big enough for a
worst case chunk of data. Only your app should access that partition,
other apps can use the dynamic part of the disk. This partition is
essentially a preallocated file and can have a normal file system on
top of it so your current app will work just fine. And the reservation
allocation is done once at partition making time, after that file
creation and updating if normal disk fast.

Regards, Steve

[email]nospamcalfee@yahoo.com[/email] (Steve Calfee) writes:[color=blue]
> On 11 Jan 2007 20:24:05 GMT, [email]phil-news-nospam@ipal.net[/email] wrote:[color=green]
>>In comp.os.linux.embedded Allan Stirrett <astirret@tandbergtv.com> wrote:
>>| To that end, we need a method to reliably and portably have the file
>>| system reserve those blocks for us when we START. However, I'm getting
>>| the impression that currently it can't be done in a timely-enough
>>| fashion for files as large as we're working with.
>>
>>Writing out commit blocks to disk is certainly not instant, but it is
>>a lot faster than many networks. Just write some zeros with dd and time
>>it to see how long.
>>
>>Another alternative is to sufficient partition the disk and ensure that
>>only a certain program is allowed to write in that space. That program
>>then manages the space numerically to reserve space rather than using the
>>filesystem to do so. It will need to also know when the space is released
>>so it can increment the available number back up.[/color]
>
> In an embedded system that must not fail, most people grab all the
> resources their app needs at once. If it fails to get them, it would
> fail eventually anyway ... better to fail quickly.[/color]

That a certain amount of disk space is not available now does not mean
it will not be available after some time has passed (especially, after
a time that is large enough to be measured in hours), so 'it would
fail eventually, anyway' is wrong. An application running on a
multiprogramming (and eventually, multiuser) system can only reliably
determine that a ressource is not available NOW by trying to acquire
it and getting an error instead of a ressource and this condition can
change at any time after the application has tried.