Metadata Server ? - Storage

This is a discussion on Metadata Server ? - Storage ; Hi, does anybody know if it's possible to have a kind of metadata server ? I explain, with 25Tb of data and many directories of 100 000 files, when i try "ls -all" it's too long ..... I'd enough time ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: Metadata Server ?

  1. Metadata Server ?

    Hi,
    does anybody know if it's possible to have a kind of metadata server ?

    I explain, with 25Tb of data and many directories of 100 000 files,
    when i try "ls -all"
    it's too long .....
    I'd enough time to take a coffee

    So i would like to find a kind of metadata server synchronize with the
    nas file system.

    I think use a virtual distributed file system like Lustre or Gfarm.


    thanks in advance,

    Séb.

  2. Re: Metadata Server ?

    On Fri, 19 Oct 2007 16:19:20 +0200, Sebastien Campion
    wrote:

    >Hi,
    >does anybody know if it's possible to have a kind of metadata server ?
    >
    >I explain, with 25Tb of data and many directories of 100 000 files,
    >when i try "ls -all"
    >it's too long .....
    >I'd enough time to take a coffee
    >
    >So i would like to find a kind of metadata server synchronize with the
    >nas file system.
    >
    >I think use a virtual distributed file system like Lustre or Gfarm.
    >
    >
    >thanks in advance,
    >
    >Séb.


    If the file system supports a seperation of data and metadata, like
    SAMFs or Lustre, then you actually *have* to have a metadata server.
    If the fs does not support it then you can't have one, at least not
    without doing some very unsupported things.

    ~F

  3. Re: Metadata Server ?

    Thanks, i learned more about the filesystem, it's WAFL by NetApp.
    In documentation, i see "The WAFL file system store metadata in files,
    three files :
    - inode file
    - block-map file
    - inode map"

    But i don't know where i can find it.

    I'd only NFS (or CIFS) protocol enable (it's a NAS NetApp) so i don't
    know if it's feasible to connect Lustre on NFS.
    If it's ok, i will install Lustre.

    Séb.


    But i don't know where

    Faeandar wrote:
    > On Fri, 19 Oct 2007 16:19:20 +0200, Sebastien Campion
    > wrote:
    >
    >
    >> Hi,
    >> does anybody know if it's possible to have a kind of metadata server ?
    >>
    >> I explain, with 25Tb of data and many directories of 100 000 files,
    >> when i try "ls -all"
    >> it's too long .....
    >> I'd enough time to take a coffee
    >>
    >> So i would like to find a kind of metadata server synchronize with the
    >> nas file system.
    >>
    >> I think use a virtual distributed file system like Lustre or Gfarm.
    >>
    >>
    >> thanks in advance,
    >>
    >> Séb.
    >>

    >
    > If the file system supports a seperation of data and metadata, like
    > SAMFs or Lustre, then you actually *have* to have a metadata server.
    > If the fs does not support it then you can't have one, at least not
    > without doing some very unsupported things.
    >
    > ~F
    >


  4. Re: Metadata Server ?

    On 2007-10-22, Sebastien Campion wrote:
    >
    > I'd only NFS (or CIFS) protocol enable (it's a NAS NetApp) so i don't
    > know if it's feasible to connect Lustre on NFS.
    > If it's ok, i will install Lustre.


    No, lustre is a completely different file system. It can't help your NetApp.

    100.000 files in each directory is going to be painfull to list,
    no matter what file system you're using. Things that might help:

    - run a name service cache daemon (nscd) to cache uid/gid lookups
    - don't resolve uid/gid by using "ls -ln"
    - use "locate" as your "metadata server", if it contains enough
    metadata for your needs.
    - see if an fscache/cachefs can help
    http://people.redhat.com/~dhowells/fscache/FS-Cache.pdf
    - tune your NFS client http://www.netapp.com/library/tr/3183.pdf


    -jf

+ Reply to Thread