everything is a directory - Plan9

This is a discussion on everything is a directory - Plan9 ; I just read this great article: http://www.linux.com/articles/114027 This quote says it all: "By the time you finish reading this article, you will be able to take advantage of the coolest file system feature to be implemented since the symbolic link. ...

+ Reply to Thread
Page 2 of 4 FirstFirst 1 2 3 4 LastLast
Results 21 to 40 of 73

Thread: everything is a directory

  1. Re: [9fans] everything is a directory

    I just read this great article: http://www.linux.com/articles/114027

    This quote says it all: "By the time you finish reading this article,
    you will be able to take advantage of the coolest file system feature
    to be implemented since the symbolic link. - CLI Magic: Use Extended
    Attributes for better file management By Ryan Paul"

    So, I guess he's right. We need to get this in to plan 9. The system
    calls should not be too hard to add, and then we can bring the
    programs over. Anyone want to do it? We could even implement the
    symlinks as a kind of extended attribute.

    ron

  2. Re: [9fans] everything is a directory

    /usr/include/sys/xattr.h (on linux anyway)
    man setxattr

    On FreeBSD it's /usr/include/sys/extattr.h

    What Unix system are you using?

    On 8/17/07, Douglas A. Gwyn wrote:
    >
    > What do you mean by "extended attributes"?
    > I haven't noticed them on the Unix systems I use.
    >



  3. Re: [9fans] everything is a directory

    I wonder what happened to my request for a UTF encoding for sarcasm...

    On 8/17/07, ron minnich wrote:
    >
    > I just read this great article: http://www.linux.com/articles/114027
    >
    > This quote says it all: "By the time you finish reading this article,
    > you will be able to take advantage of the coolest file system feature
    > to be implemented since the symbolic link. - CLI Magic: Use Extended
    > Attributes for better file management By Ryan Paul"
    >
    > So, I guess he's right. We need to get this in to plan 9. The system
    > calls should not be too hard to add, and then we can bring the
    > programs over. Anyone want to do it? We could even implement the
    > symlinks as a kind of extended attribute.
    >
    >
    > ron
    >



  4. Re: [9fans] everything is a directory

    > I wonder what happened to my request for a UTF encoding for sarcasm...

    0x263A


  5. Re: [9fans] everything is a directory

    On 8/17/07, David Leimbach wrote:
    > I wonder what happened to my request for a UTF encoding for sarcasm...


    OK, you win the prize! I am a lousy troll ...

    ron

  6. Re: [9fans] everything is a directory

    On 8/17/07, ron minnich wrote:
    >
    > On 8/17/07, David Leimbach wrote:
    > > I wonder what happened to my request for a UTF encoding for sarcasm...

    >
    > OK, you win the prize! I am a lousy troll ...
    >
    > ron
    >


    Nah I just thought you were joking about the article because the author
    starts by singing the praises of the symbolic link. :-)

    My objections to their well established existence are based on feeling
    uneasy about the answers to the following questions:

    What happens to file metadata that you
    1. Put on NFS?
    2. Access with 9p?
    3. Copy from one filesystem to another?
    4. Copy from one OS to another?
    5. Archive one one and move to another OS then extract? (smart archiver?)

    Somehow I don't feel like I need this feature to do anything and that any
    data I put in there is destined to be lost on backup and restore which is
    part of why I worked on xar to do extended attributes for Mac OS X... I
    really wanted to be able to backup Microsoft Office and have it restore and
    actually work, "tar" couldn't do this, "pax" couldn't do this (well at the
    time anyway they might now....).

    See for yourself, there's at least 3 implementations of extended attribute
    "backends" in xar.

    (fbsdattr, darwinattr, linuxattr)
    http://xar.googlecode.com/svn/trunk/xar/lib/


    Like em or not, they're here... and we have to deal. I just don't want to
    see Plan 9 adopt them because it seems possible to do everything they enable
    us to do by fixing file formats. Clearly I can design one format that
    archives them from 3 different OSes (and yeah, we used XML for that cuz
    everything is better with XML!!!).

    At any rate, it certainly increases job security for those who know about
    them and how to use em! :-)

    Dave






    Dave


  7. Re: [9fans] everything is a directory

    > At any rate, it certainly increases job security for those who know about
    > them and how to use em! :-)


    just blame the stock market crash, errr... `correction'
    on extended attributes, in a blog somewhere
    and soon there will be hearings about them.


  8. Re: [9fans] everything is a directory

    ron minnich wrote:
    > On 8/17/07, Douglas A. Gwyn wrote:
    > > What do you mean by "extended attributes"?
    > > I haven't noticed them on the Unix systems I use.

    > maybe I'm missing the question, but on my linux:
    > man -k extended | grep attrib | wc
    > 21 183 1417


    On Solaris 8 there is only one line, for user_attr(4),
    which pertains to users, not to files.
    However, on Solaris 10 it turns up fsattr(5),
    which seems to be the feature under discussion.
    The Solaris 10 manual page for fsattr(5) describes
    nicely how to access the attribute information, but
    it fails to explain why one would want to use that
    feature in the first place.

    > it's everywhere. But it was too hard to put in in some normal way, so
    > it went in from the side.


    At least on Solaris 10, there is an openat() syscall
    that opens the attributes as a normal fd starting at
    the (possibly ordinary) fd supplied as an argument,
    which acts "sort of" like a current-working-directory.

    I'm still wondering what the motivation was.

  9. Re: everything is a directory

    On Aug 17, 5:06 am, quans...@quanstro.net (erik quanstrom) wrote:
    > > They already have synthetic file systems built into NT!

    >
    > i think it's worse thank that. attributes and their ilk essentially
    > add methods to a filesystem.


    That's overstatement -- attributes add 'user defined' types to the
    filesystem, but that's not the same thing as giving each filesystem
    object procedures. It might require polymorphic versions of ls, cat,
    &c. -- but probably not, since the extra fields aren't of interest to
    them.

    I've seen a lot of criticism of extended attributes on this thread,
    but no one has stepped up with a solution that addresses the problem
    they solve. Application specific data should go in the file -- we all
    agree about that. The file's position in the heirarchy is modeled by
    directories, which also carry OS specific metadata -- permissions,
    ownership. But where do the oddball intermediaries put their metadata?
    For example, where should the desktop environment put file specific
    icons or annotations? It can't very well stuff icons into your
    doom .wad files. In principle, the desktop could put that stuff in a
    mockup of the real file-system hierarchy, but having a /gnome/icons
    tree that had to be kept in synchrony with the 'real' filesystem
    invites a lot of semantic confusion, as well as an implementation
    hassle (broken links, new versions of 'mv', something else?).

    The fathers of Unix saw many things, but who's to say they saw all the
    metadata we will ever need?

  10. Re: [9fans] everything is a directory

    "erik quanstrom" wrote in message
    news:198ff93d3223123a0e103cbcc4755bc4@quanstro.net ...
    > hey, how do you grep for stuff in extended attributes?


    grep -@, assuming that grep has been extended like other utilities (tar,
    etc.)

    One would have thought that a better design would have been
    /path/file/.attributes/...
    Who makes these decisions, anyway??

  11. Re: [9fans] Re: everything is a directory

    Do you prefer embedding the "face" in each mail as metadata instead of
    keeping a faces db?

    I suffer the same thing with mp3 art, embedded in the mp3s. Lots of dup. data
    plus some players that cannot cope with such decorated mp3 files,
    including the one
    in my car

    On 8/20/07, jsnx wrote:
    > On Aug 17, 5:06 am, quans...@quanstro.net (erik quanstrom) wrote:
    > > > They already have synthetic file systems built into NT!

    > >
    > > i think it's worse thank that. attributes and their ilk essentially
    > > add methods to a filesystem.

    >
    > That's overstatement -- attributes add 'user defined' types to the
    > filesystem, but that's not the same thing as giving each filesystem
    > object procedures. It might require polymorphic versions of ls, cat,
    > &c. -- but probably not, since the extra fields aren't of interest to
    > them.
    >
    > I've seen a lot of criticism of extended attributes on this thread,
    > but no one has stepped up with a solution that addresses the problem
    > they solve. Application specific data should go in the file -- we all
    > agree about that. The file's position in the heirarchy is modeled by
    > directories, which also carry OS specific metadata -- permissions,
    > ownership. But where do the oddball intermediaries put their metadata?
    > For example, where should the desktop environment put file specific
    > icons or annotations? It can't very well stuff icons into your
    > doom .wad files. In principle, the desktop could put that stuff in a
    > mockup of the real file-system hierarchy, but having a /gnome/icons
    > tree that had to be kept in synchrony with the 'real' filesystem
    > invites a lot of semantic confusion, as well as an implementation
    > hassle (broken links, new versions of 'mv', something else?).
    >
    > The fathers of Unix saw many things, but who's to say they saw all the
    > metadata we will ever need?
    >


  12. Re: [9fans] Re: everything is a directory

    > The fathers of Unix saw many things, but who's to say they saw all the
    > metadata we will ever need?


    who's to say they didn't. (i hope we're talking about plan 9 here, not
    unix. in plan 9 many cumbersome traditions were broken. if
    they had wanted attributes, they would have added them.)

    i don't see a specific problem in plan 9 that extra metadata solves.
    nemo makes a great point. the only place that plan 9 has icons, they
    are kept in /lib/face as files.

    - erik

  13. Re: [9fans] everything is a directory

    I was thinking more about the subject, and extended attributes.
    Here is an alternative proposal to implement extended attributes:

    (1) Everything stays the same as it is (Plan 9 or old-Unix)
    except:
    (2) An attempt to read(2) or write(2) a directory will succeed
    if and only if there is a .data entry, in which case it will
    access contents of the .data object.

    None of the existing tools need to be changed.

    To add attributes (all done in user space):
    (a) Move the current file (terminal node) to a temp name.
    (b) Make a directory in its place, with the original name.
    (c) Move the temp name to original_name/.data .
    (d) Create a file (subdirectory?) .attributes .
    (e) Populate the .attributes .

  14. Re: [9fans] everything is a directory

    But why don't just keep a foo.attr side by side with the file then?
    Should you need that, I mean.

    On 8/20/07, Douglas A. Gwyn wrote:
    > I was thinking more about the subject, and extended attributes.
    > Here is an alternative proposal to implement extended attributes:
    >
    > (1) Everything stays the same as it is (Plan 9 or old-Unix)
    > except:
    > (2) An attempt to read(2) or write(2) a directory will succeed
    > if and only if there is a .data entry, in which case it will
    > access contents of the .data object.
    >
    > None of the existing tools need to be changed.
    >
    > To add attributes (all done in user space):
    > (a) Move the current file (terminal node) to a temp name.
    > (b) Make a directory in its place, with the original name.
    > (c) Move the temp name to original_name/.data .
    > (d) Create a file (subdirectory?) .attributes .
    > (e) Populate the .attributes .
    >


  15. Re: [9fans] everything is a directory

    This is only possible because someone said it was OK to
    break directories in the first place. Not too long ago, if you
    wanted to read a directory you could use a read system call.
    I see no functional reason that that symmetry was worth
    breaking.

    Also, I dispute your claim that tools have to change.
    For instance, ls will misbehave: it will think your file
    is a directory. And there will be lots more instances
    because, fundamentally, stat will be misleading.
    Then all the existing tools will be changed because
    people will want to use them on attributed files and
    their attributes.

    It reminds me of symlinks, which I detest. I understand
    why they were created - cross-device links didn't work -
    but they broke just about everything by introducing a
    second kind of file. Half the time you want the file,
    half the time you want the file it describes. They're
    pointers, and deciding whether the operation wants the
    pointer's value or referent means a new system call
    shows up, a new flag appears everywhere, and one
    usually guesses wrong the first time.

    Attributes, as created or as posited here, are better
    only because for the most part one can ignore them
    altogether. Except when you can't.

    If you want attributes, rethink the system, don't hack
    it.

    -rob



    On 8/20/07, Douglas A. Gwyn wrote:
    > I was thinking more about the subject, and extended attributes.
    > Here is an alternative proposal to implement extended attributes:
    >
    > (1) Everything stays the same as it is (Plan 9 or old-Unix)
    > except:
    > (2) An attempt to read(2) or write(2) a directory will succeed
    > if and only if there is a .data entry, in which case it will
    > access contents of the .data object.
    >
    > None of the existing tools need to be changed.
    >
    > To add attributes (all done in user space):
    > (a) Move the current file (terminal node) to a temp name.
    > (b) Make a directory in its place, with the original name.
    > (c) Move the temp name to original_name/.data .
    > (d) Create a file (subdirectory?) .attributes .
    > (e) Populate the .attributes .
    >


  16. Re: [9fans] Re: everything is a directory

    On Mon, 2007-08-20 at 08:47 +0000, jsnx wrote:
    > On Aug 17, 5:06 am, quans...@quanstro.net (erik quanstrom) wrote:
    > > > They already have synthetic file systems built into NT!

    > >
    > > i think it's worse thank that. attributes and their ilk essentially
    > > add methods to a filesystem.

    >
    > That's overstatement -- attributes add 'user defined' types to the
    > filesystem, but that's not the same thing as giving each filesystem
    > object procedures. It might require polymorphic versions of ls, cat,
    > &c. -- but probably not, since the extra fields aren't of interest to
    > them.
    >
    > I've seen a lot of criticism of extended attributes on this thread,
    > but no one has stepped up with a solution that addresses the problem
    > they solve. Application specific data should go in the file -- we all
    > agree about that. The file's position in the heirarchy is modeled by
    > directories, which also carry OS specific metadata -- permissions,
    > ownership. But where do the oddball intermediaries put their metadata?


    If that's the only need that can justify extended file attributes my
    personal reaction would be to quote Al Viro (who, I believe, was
    paraphrasing Ken, but I'm not certain): "Out-of-band == should be on a
    separate channel..."

    To me extended attributes are no better than ioctls and I hope we
    all share a certain deep and profound feeling towards them.

    One last observation. I would argue that the very same reason that
    makes symlinks semi-interisting on UNIX-like systems makes extended
    attributes pop-up in the same context as well: it is difficult to
    manipulate your personal file namespace there and create these
    "separate channels" on demand.

    Thanks,
    Roman.


  17. Re: [9fans] Re: everything is a directory

    > If that's the only need that can justify extended file attributes my
    > personal reaction would be to quote Al Viro (who, I believe, was
    > paraphrasing Ken, but I'm not certain): "Out-of-band == should be on a
    > separate channel..."


    I would love to find the original quote from ken (the one from Al is
    quite good though). I personally like to put it this way:

    "Metadata is just data with a stupid interface."

    > To me extended attributes are no better than ioctls and I hope we
    > all share a certain deep and profound feeling towards them.


    Very true.

    > One last observation. I would argue that the very same reason that
    > makes symlinks semi-interisting on UNIX-like systems makes extended
    > attributes pop-up in the same context as well: it is difficult to
    > manipulate your personal file namespace there and create these
    > "separate channels" on demand.


    Sad thing is that when you try to convince Unix people of the power of
    private namespaces, they complain that they seem 'too confusing'...
    yea, because symlinks, extended attributes and other such hideous
    hacks are so much nicer... and certainly create many more jobs, so
    they keep adding layers of crud but wont implement the solution that
    has been around for ages... has anyone learned anything from plan9?
    I'm starting to doubt it.

    Why is it that every new software feature people come up with makes me
    think of the "I did it for you all" interview?

    uriel

  18. Re: everything is a directory

    On Aug 20, 7:55 am, n...@lsub.org (Francisco J Ballesteros) wrote:
    > But why don't just keep a foo.attr side by side with the file then?


    That would poison the file namespace.

  19. Re: everything is a directory

    On Aug 20, 1:48 am, "Douglas A. Gwyn" wrote:
    > "erik quanstrom" wrote in message
    >
    > news:198ff93d3223123a0e103cbcc4755bc4@quanstro.net ...
    >
    > > hey, how do you grep for stuff in extended attributes?

    >
    > grep -@, assuming that grep has been extended like other utilities (tar,
    > etc.)
    >
    > One would have thought that a better design would have been
    > /path/file/.attributes/...
    > Who makes these decisions, anyway??


    Putting attributes in the same namespace as files introduces a whole
    new problem -- just as storing ownership in foo/.owners would.

  20. Re: [9fans] everything is a directory

    "Rob Pike" wrote in message
    news:7359f0490708200802g2ba8abc0jd61b4f9550bc382f@ mail.gmail.com...
    > This is only possible because someone said it was OK to
    > break directories in the first place. Not too long ago, if you
    > wanted to read a directory you could use a read system call.
    > I see no functional reason that that symmetry was worth
    > breaking.


    That was already changed long before "extended attributes"
    entered the picture. On modern Unix systems (since NFS
    or slightly before) and even on Plan 9 one doesn't normally
    read and parse actual directory content; the OS/library has
    a different interface for directory interrogation.

    > Also, I dispute your claim that tools [don't] have to change.
    > For instance, ls will misbehave: it will think your file
    > is a directory.


    But under my proposal that is correct. An attributed file
    *is* an ordinary directory (with ordinary files and/or other
    directories as members). But if you "cat" or "grep" the file
    name, you access its .data member. Since these days the
    actual directory content isn't meaningfully accessed, that frees
    it up for some other interpretation, such as what I proposed.
    (You can also access the file data as path/name/.data, so
    there is nothing misleading about this use of the existing
    filesystem structure.)

    (An implementation detail I didn't mention: I recommend
    that the .data member always be the first entry for speed
    of access, thus no "." or ".." entries [unlike Solaris-10
    extended attributes]; those should be handled by the OS
    path interpreter, as on Plan 9.)

    > And there will be lots more instances
    > because, fundamentally, stat will be misleading.
    > Then all the existing tools will be changed because
    > people will want to use them on attributed files and
    > their attributes.


    There is no reason why "attribute aware" tools shouldn't
    be developed, if attributes are indeed useful (something
    I'm not yet convinced of). My purpose was to figure
    out a way of implementating extended file attributes with
    minimal disruption to the usual Unix/Plan 9 file model.
    Note that the Linux/Solaris-10 (Posix?) implementation
    requires separate system calls and extensive changes to
    standard tools *and* to the way those tools are invoked.

    > It reminds me of symlinks, which I detest. ...


    Yes, the analogy is close.

    > but they broke just about everything by introducing a
    > second kind of file. ...


    Which is what we want to avoid this time.

    > Attributes, as created or as posited here, are better
    > only because for the most part one can ignore them
    > altogether. Except when you can't.


    Unfortunately, if you forget about attributes on Linux/
    Solaris-10, you far too easily do what users would
    definitely consider to be "the wrong thing". That's
    *why* so many standard tools had to be modified
    using their approach to extended attributes.

    > If you want attributes, rethink the system, don't hack
    > it.


    So what is a better proposal?

+ Reply to Thread
Page 2 of 4 FirstFirst 1 2 3 4 LastLast