File size limitations - Storage

This is a discussion on File size limitations - Storage ; Is anyone aware of a way to limit the size of files allowed on a file system? Specifically, I don't want files to be able to grow past XGB. Thanks. ~F...

+ Reply to Thread
Results 1 to 12 of 12

Thread: File size limitations

  1. File size limitations

    Is anyone aware of a way to limit the size of files allowed on a file
    system?

    Specifically, I don't want files to be able to grow past XGB.

    Thanks.

    ~F

  2. Re: File size limitations

    Faeandar wrote:
    > Is anyone aware of a way to limit the size of files allowed on a file
    > system?
    >
    > Specifically, I don't want files to be able to grow past XGB.


    Varies somewhat from filesystem to filesystem, are you looking for the
    limits of a particular filesystem (eg. NTFS, EXT# etc) Or are you
    looking for a comparison of various file systems, or a filesystem thht
    can handle a specific file size (if so you need to be more specific than
    XGB ;-)

    --
    Nik Simpson

  3. Re: File size limitations

    On Tue, 30 Oct 2007 17:04:38 -0500, nik Simpson
    wrote:

    >Faeandar wrote:
    >> Is anyone aware of a way to limit the size of files allowed on a file
    >> system?
    >>
    >> Specifically, I don't want files to be able to grow past XGB.

    >
    >Varies somewhat from filesystem to filesystem, are you looking for the
    >limits of a particular filesystem (eg. NTFS, EXT# etc) Or are you
    >looking for a comparison of various file systems, or a filesystem thht
    >can handle a specific file size (if so you need to be more specific than
    >XGB ;-)



    Specifically WAFL and I want to *LIMIT* the file size, not determine
    the max. I use XGB because it will vary by file system (all WAFL but
    many of them). But here's an example:

    for vol1 I want to limit *any* file size to 35GB
    for vol4 I want to limit *any* file size to 20GB

    etc.

    Thanks.

    ~F

  4. Re: File size limitations

    Faeandar wrote:

    I want to *LIMIT* the file size, not determine
    > the max. I use XGB because it will vary by file system (all WAFL but
    > many of them). But here's an example:
    >
    > for vol1 I want to limit *any* file size to 35GB
    > for vol4 I want to limit *any* file size to 20GB
    >
    > etc.


    I've never heard of that specific control - and am kind of curious what
    you need it for (new feature requests being a topic I'm interested in).

    - bill

  5. Re: File size limitations

    On Tue, 30 Oct 2007 18:23:58 -0500, Bill Todd
    wrote:

    >Faeandar wrote:
    >
    > I want to *LIMIT* the file size, not determine
    >> the max. I use XGB because it will vary by file system (all WAFL but
    >> many of them). But here's an example:
    >>
    >> for vol1 I want to limit *any* file size to 35GB
    >> for vol4 I want to limit *any* file size to 20GB
    >>
    >> etc.

    >
    >I've never heard of that specific control - and am kind of curious what
    >you need it for (new feature requests being a topic I'm interested in).
    >
    >- bill



    We know that, for a given set of file systems, any file greater than X
    is unsuable by the apps but we still get runaway processes that create
    files 2, 3, sometimes even 5 times X. We find these files only when
    we run SRM walks and we are finding them more frequently than I would
    have thought.

    So in an effort to reduce storage usage and save other people from
    running out of space I want to limit the file size to X+25%. I want
    the extra 25% so that audits will still pick up bogus files but they
    do not bork the entire file system.

    Are you just generally interested in feature requests or is it part of
    your job?

    ~F

  6. Re: File size limitations

    Faeandar wrote:

    ....

    > Are you just generally interested in feature requests or is it part of
    > your job?


    Sort of both, since most of my work for the past couple of decades has
    involved creating new file systems and even when I'm not actively
    engaged in doing so it's good to have an idea what might want to be
    included in the next one.

    - bill

  7. Re: File size limitations

    In article <42hfi3hism08nnen2bsss9vf9tli9tkp3g@4ax.com>,
    Faeandar wrote:
    >
    >We know that, for a given set of file systems, any file greater than X
    >is unsuable by the apps but we still get runaway processes that create
    >files 2, 3, sometimes even 5 times X. We find these files only when
    >we run SRM walks and we are finding them more frequently than I would
    >have thought.
    >
    >So in an effort to reduce storage usage and save other people from
    >running out of space I want to limit the file size to X+25%. I want
    >the extra 25% so that audits will still pick up bogus files but they
    >do not bork the entire file system.


    This is the first time I hear of such a requirement, and I don't know
    of any file system that has this functionality NOW, but there very
    well might be some, so please call around.

    If you want to try to get vendors to add this functionality, a good
    starting point would be in two places. First, look for a suite of
    storage management tools that has built-in SRM, and that are tightly
    integrated with file systems. It should be relatively easy for the
    management application to set alarms on file sizes, and react to
    alarms by making the file unwriteable. Second, look for file systems
    that have policy-based control of where the data is stored. They
    could (theoretically) be enhanced so the policy is evaluated during
    write time, which would allow you to write a policy as follows:
    - If file size <10GB, write file data to disk X.
    - If file size >10GB, write file data to /dev/null, which will
    immediately run out of space (it has zero blocks free), so writing
    will stop.

    In theory, it is possible to enhance quota management systems to have
    a file size quota; but that would be a long row to hoe.

    Don't forget: If your solution uses NAS boxes (whether they be NFS
    V2/3/4 or CIFS), the file system is inside the NAS box, not on the
    client, so you would have to talk to the server vendor.

    Good luck!

    --
    Ralph Becker-Szendy _firstname_@lr_dot_los-gatos_dot_ca_dot_us
    735 Sunset Ridge Road; Los Gatos, CA 95033

  8. Re: File size limitations

    On Wed, 31 Oct 2007 16:35:38 -0000,
    _firstname_@lr_dot_los-gatos_dot_ca_dot_us wrote:

    >In article <42hfi3hism08nnen2bsss9vf9tli9tkp3g@4ax.com>,
    >Faeandar wrote:
    >>
    >>We know that, for a given set of file systems, any file greater than X
    >>is unsuable by the apps but we still get runaway processes that create
    >>files 2, 3, sometimes even 5 times X. We find these files only when
    >>we run SRM walks and we are finding them more frequently than I would
    >>have thought.
    >>
    >>So in an effort to reduce storage usage and save other people from
    >>running out of space I want to limit the file size to X+25%. I want
    >>the extra 25% so that audits will still pick up bogus files but they
    >>do not bork the entire file system.

    >
    >This is the first time I hear of such a requirement, and I don't know
    >of any file system that has this functionality NOW, but there very
    >well might be some, so please call around.
    >
    >If you want to try to get vendors to add this functionality, a good
    >starting point would be in two places. First, look for a suite of
    >storage management tools that has built-in SRM, and that are tightly
    >integrated with file systems. It should be relatively easy for the
    >management application to set alarms on file sizes, and react to
    >alarms by making the file unwriteable. Second, look for file systems
    >that have policy-based control of where the data is stored. They
    >could (theoretically) be enhanced so the policy is evaluated during
    >write time, which would allow you to write a policy as follows:
    >- If file size <10GB, write file data to disk X.
    >- If file size >10GB, write file data to /dev/null, which will
    > immediately run out of space (it has zero blocks free), so writing
    > will stop.
    >
    >In theory, it is possible to enhance quota management systems to have
    >a file size quota; but that would be a long row to hoe.
    >
    >Don't forget: If your solution uses NAS boxes (whether they be NFS
    >V2/3/4 or CIFS), the file system is inside the NAS box, not on the
    >client, so you would have to talk to the server vendor.
    >
    >Good luck!


    Any hints or suggestions as to who these 3rd party vendors might be?
    That specifically manage data to/from disk?

    Thanks.

    ~F

  9. Re: File size limitations

    Faeandar wrote:
    > Is anyone aware of a way to limit the size of files allowed on a file
    > system?
    >
    > Specifically, I don't want files to be able to grow past XGB.
    >
    > Thanks.
    >
    > ~F


    Wouldn't quota management software be an option?
    No personal experience though.

  10. Re: File size limitations

    On Fri, 02 Nov 2007 09:22:35 +0100, Benno
    wrote:

    >Faeandar wrote:
    >> Is anyone aware of a way to limit the size of files allowed on a file
    >> system?
    >>
    >> Specifically, I don't want files to be able to grow past XGB.
    >>
    >> Thanks.
    >>
    >> ~F

    >
    >Wouldn't quota management software be an option?
    >No personal experience though.


    Quota management is based on user id or file system/directory, not the
    file itself.

    I may have a user who needs to exceed 20GB in total but not in a
    single file. Today's quota software makes no distinction between the
    two.

    ~F

  11. Re: File size limitations

    Faeandar wrote:
    > On Fri, 02 Nov 2007 09:22:35 +0100, Benno
    > wrote:
    >
    >> Faeandar wrote:
    >>> Is anyone aware of a way to limit the size of files allowed on a file
    >>> system?
    >>>
    >>> Specifically, I don't want files to be able to grow past XGB.
    >>>
    >>> Thanks.
    >>>
    >>> ~F

    >> Wouldn't quota management software be an option?
    >> No personal experience though.

    >
    > Quota management is based on user id or file system/directory, not the
    > file itself.
    >
    > I may have a user who needs to exceed 20GB in total but not in a
    > single file. Today's quota software makes no distinction between the
    > two.
    >
    > ~F


    Hi!

    A simple ulimit on unix server does what you want, but it's a per user
    id entry and i dont know if it is available over network (target, not
    source user id based).

    with best regards
    Dieter Stumpner

  12. Re: File size limitations

    On Oct 30, 3:25 pm, Faeandar wrote:
    > Specifically WAFL and I want to *LIMIT* the file size, not determine
    > the max. I use XGB because it will vary by file system (all WAFL but
    > many of them). But here's an example:
    >
    > for vol1 I want to limit *any* file size to 35GB
    > for vol4 I want to limit *any* file size to 20GB
    >
    > etc.
    >
    > Thanks.
    >
    > ~F


    Since you mentioned WAFL, have you looked at their FPOLICY
    implementation? There are a few 3rd party software that uses this API
    for various file level manipulation though I haven't seen any that
    specifically does for file size but have seen some that prevent files
    with certain extensions from being stored on specific qtree or volumes
    for example. But I think it could be doable. NetApp may have a list of
    3rd parties that use their FPOLICY API
    -G


+ Reply to Thread