Faking dense files... - Linux

This is a discussion on Faking dense files... - Linux ; To all, Thank you again for all of the ideas. Perhaps this will give a clear picture of what I had in mind. This article describes a patch for ext3 that allows for block reservation. http://lwn.net/Articles/81357/ It doesn't look like ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 25 of 25

Thread: Faking dense files...

  1. Re: Faking dense files...

    To all,

    Thank you again for all of the ideas. Perhaps this will give a
    clear picture of what I had in mind. This article describes a patch
    for ext3 that allows for block reservation.

    http://lwn.net/Articles/81357/

    It doesn't look like this made it into the main ext3 code, but
    perhaps for ext4... While it sounds from the article like it's not
    completely "tight" ("...will not actually prevent blocks inside the
    window from being allocated to some other file"), this is the kind of
    thing I was originally asking about. In particular, since it appears
    to be done in core without any disk I/O, it should be very quick, even
    for very large files.

    So I'm getting the feeling that what I'm asking about (quick,
    in-core block reservation) is certainly not standard or even available
    in the form I'm looking for currently. However, I'll keep my eyes
    peeled for ext4 and/or other file systems that have this feature, and,
    when it shows up, use that as the recommended file system for installs
    of my app. While this won't cover all systems, it might be the best I
    can do. Thank you all again for the input.

    Allan Stirrett


  2. Re: Faking dense files...

    phil-news-nospam@ipal.net writes:
    > In comp.os.linux.embedded Rainer Weikusat wrote:


    [...]

    > | I understand what he would like to do, but that doesn't change the
    > | fact that this is basically a policy-descision made by someone using
    > | the respective applications. That someone needs to have enough disk
    > | space available for all concurrently running tasks and if there isn't
    > | enough disk space available, some or all of them will fail and the
    > | solution is to either not have different applications compete for
    > | resources that are scarce enough that there will be a 'loser' and a
    > | 'winner' or to accept that this will happen.
    >
    > Competing for resource is not an uncommon issue. But such competition
    > does operate more gracefully if the latter tasks can at least know BEFORE
    > they take certain steps, that the space is committed to them.


    This statement boils down to 'not competing for ressources is more
    graceful than competing for ressources'. But if application A is so
    much more important than application B that the latter should always
    fail if the former is active, why the heck are they running
    concurrently and compete for ressources in the first place?

    Judging from the other end: If application A needs several hours to
    complete, why should it voluntarily refuse to work because of a
    condition that was true at the time someone wanted it to work, but may
    no longer be true several hours later?

  3. Re: Faking dense files...

    Hello,

    Allan Stirrett wrote:

    > Thank you again for all of the ideas. Perhaps this will give a
    > clear picture of what I had in mind. This article describes a patch
    > for ext3 that allows for block reservation.
    >
    > http://lwn.net/Articles/81357/


    Besides that this reservation was not supposed to be for large amounts,
    only for relatively small numbers of blocks to avoid fragmentation, and
    it was meant to be done without user interaction.

    > So I'm getting the feeling that what I'm asking about (quick,
    > in-core block reservation) is certainly not standard or even available
    > in the form I'm looking for currently.


    Reserving large amounts of blocks is strictly contrary to the goals of
    an OS. This can only fail, because if every app always uses the upper
    bound for reservation before any actual operation, resources will seem
    exhausted a lot earlier before anything is really exhausted.

    If you used more and smaller files instead of one big, you could more
    easily spread over more discs avoiding the race. Maybe it could be
    arranged more easily that the initial files could be consumed before
    the later files fill the space, e.g. you could wait for space to become
    available for the later files. If there is a point in time, where you
    need more disk space than available, then no trick will help you over
    the fact that you need more discs.


    > However, I'll keep my eyes
    > peeled for ext4 and/or other file systems that have this feature, and,
    > when it shows up, use that as the recommended file system for installs
    > of my app. While this won't cover all systems, it might be the best I
    > can do. Thank you all again for the input.


    This is no filesystem problem, it is an application problem to handle
    resources properly, and an administration problem to provide enough
    resources.

    Bernd Strieder


  4. Re: Faking dense files...

    On 11 Jan 2007 20:24:05 GMT, phil-news-nospam@ipal.net wrote:

    >In comp.os.linux.embedded Allan Stirrett wrote:
    >
    >| To that end, we need a method to reliably and portably have the file
    >| system reserve those blocks for us when we START. However, I'm getting
    >| the impression that currently it can't be done in a timely-enough
    >| fashion for files as large as we're working with.
    >
    >Writing out commit blocks to disk is certainly not instant, but it is
    >a lot faster than many networks. Just write some zeros with dd and time
    >it to see how long.
    >
    >Another alternative is to sufficient partition the disk and ensure that
    >only a certain program is allowed to write in that space. That program
    >then manages the space numerically to reserve space rather than using the
    >filesystem to do so. It will need to also know when the space is released
    >so it can increment the available number back up.
    >
    >--


    In an embedded system that must not fail, most people grab all the
    resources their app needs at once. If it fails to get them, it would
    fail eventually anyway ... better to fail quickly.

    So the best solution I have seen here is to partition your disk. Make
    one partition just for your data collection app and big enough for a
    worst case chunk of data. Only your app should access that partition,
    other apps can use the dynamic part of the disk. This partition is
    essentially a preallocated file and can have a normal file system on
    top of it so your current app will work just fine. And the reservation
    allocation is done once at partition making time, after that file
    creation and updating if normal disk fast.

    Regards, Steve


  5. Re: Faking dense files...

    nospamcalfee@yahoo.com (Steve Calfee) writes:
    > On 11 Jan 2007 20:24:05 GMT, phil-news-nospam@ipal.net wrote:
    >>In comp.os.linux.embedded Allan Stirrett wrote:
    >>| To that end, we need a method to reliably and portably have the file
    >>| system reserve those blocks for us when we START. However, I'm getting
    >>| the impression that currently it can't be done in a timely-enough
    >>| fashion for files as large as we're working with.
    >>
    >>Writing out commit blocks to disk is certainly not instant, but it is
    >>a lot faster than many networks. Just write some zeros with dd and time
    >>it to see how long.
    >>
    >>Another alternative is to sufficient partition the disk and ensure that
    >>only a certain program is allowed to write in that space. That program
    >>then manages the space numerically to reserve space rather than using the
    >>filesystem to do so. It will need to also know when the space is released
    >>so it can increment the available number back up.

    >
    > In an embedded system that must not fail, most people grab all the
    > resources their app needs at once. If it fails to get them, it would
    > fail eventually anyway ... better to fail quickly.


    That a certain amount of disk space is not available now does not mean
    it will not be available after some time has passed (especially, after
    a time that is large enough to be measured in hours), so 'it would
    fail eventually, anyway' is wrong. An application running on a
    multiprogramming (and eventually, multiuser) system can only reliably
    determine that a ressource is not available NOW by trying to acquire
    it and getting an error instead of a ressource and this condition can
    change at any time after the application has tried.


+ Reply to Thread
Page 2 of 2 FirstFirst 1 2