[9fans] store 9p session-only values using lib9p - Plan9

This is a discussion on [9fans] store 9p session-only values using lib9p - Plan9 ; Hello I've a 9p server implemented using lib9p which serves decoded files, for example, i have a base64 encoded file i want to read, but i want to decode it at the same time the client reads. Then i need ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 27

Thread: [9fans] store 9p session-only values using lib9p

  1. [9fans] store 9p session-only values using lib9p

    Hello

    I've a 9p server implemented using lib9p which serves decoded files, for example, i have a base64 encoded file i want to read, but i want to decode it at the same time the client reads. Then i need to save two offsets, the one sent to the client corresponds to the decoded data, and i need other which correspond to the original data. (base64 is an example, rfc2047 codification is the main issue)

    If i save the offset adjustment in f->aux or simmilar, i can calculate the real file offset on the next T-reads, but that will only work if one client reads it at one time.

    could that value be saved for each session?

    i would love to hear from other's experiences,

    thanks,

    gabi



  2. Re: [9fans] store 9p session-only values using lib9p

    > If i save the offset adjustment in f->aux or simmilar,
    > i can calculate the real file offset on the next T-reads,
    > but that will only work if one client reads it at one time.


    No it won't. Even if just one client is reading, that client might seek.

    You could assume that most reads will pick up where the
    last one left off, and use f->aux to point at a single hint
    that you can check. Maybe you're willing to do more work
    if you do get a seek. Otherwise you'll need to build a more
    complex data structure to help you map between the two.

    Also, if f is a Fid* (not a File*), then you get a different f for each
    different Topen of the file (and thus also for each client), so the
    hint-per-Fid usually works pretty well in practice for this kind
    of thing.

    Russ



  3. Re: [9fans] store 9p session-only values using lib9p

    > Hello
    >
    > I've a 9p server implemented using lib9p which serves decoded files, for example, i have a base64 encoded file i want to read, but i want to decode it at the same time the client reads. Then i need to save two offsets, the one sent to the client corresponds to the decoded data, and i need other which correspond to the original data. (base64 is an example, rfc2047 codification is the main issue)
    >
    > If i save the offset adjustment in f->aux or simmilar, i can calculate the real file offset on the next T-reads, but that will only work if one client reads it at one time.
    >
    > could that value be saved for each session?
    >
    > i would love to hear from other's experiences,


    i've been spending a little time with upas/fs. i don't think that in
    general one can get by parsing part of an email, unless that part
    happens to be top-level headers. there are more transforms than
    you've mentioned. apple mail loves to put nulls in html email, for
    example.

    traditional mbox format is problematic, too.

    if you wish to control memory usage (which is a serious problem for
    upas/fs), i think the way to go is a cache with reference counting.

    the stuff i have isn't ready for prime time, but it's coming along
    nicely.

    by the way, does anyone know the rational for the date on the
    unix "From " line? upas sets it to the date the message is originally
    delivered to the inbox. moving it from the inbox to another folder
    does not change the date.

    - erik



  4. Re: [9fans] upas/fs

    > by the way, does anyone know the rational for the date on the
    > unix "From " line? upas sets it to the date the message is originally
    > delivered to the inbox. moving it from the inbox to another folder
    > does not change the date.


    the date is the date it was delivered.
    it's a receiver-side postmark.
    but you knew that. what's your question?

    russ



  5. Re: [9fans] upas/fs

    >> by the way, does anyone know the rationale for the date on the
    >> unix "From " line? upas sets it to the date the message is originally
    >> delivered to the inbox. moving it from the inbox to another folder
    >> does not change the date.

    >
    > the date is the date it was delivered.
    > it's a receiver-side postmark.
    > but you knew that. what's your question?


    right. since the date is attached when delivered to a mailbox,
    why doesn't this date change when it's delivered to a secondary
    mailbox? why is the assignment a magical property of the inbox?

    - erik



  6. Re: [9fans] upas/fs


    On 2008-Jun-11, at 19:31 , erik quanstrom wrote:

    > right. since the date is attached when delivered to a mailbox,
    > why doesn't this date change when it's delivered to a secondary
    > mailbox? why is the assignment a magical property of the inbox?


    Most likely it's just an artifact of the original UNIX mail
    implementation. The \n^From separator line got generated at
    initial delivery time, and the mail client used that as the display
    time in the message summaries (e.g. Date: not spoken here). Therefore
    it makes sense to preserve the initial separator line with it's date
    intact to ensure consistency for display purposes. Think of it as the
    UNIX ctime of the message.

    --lyndon


  7. Re: [9fans] store 9p session-only values using lib9p

    hello

    at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded.

    also file lengths is going to be a problem if i'm going to decode files within the fs.

    i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs

    about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-?

    thanks,

    gabi


  8. Re: [9fans] store 9p session-only values using lib9p

    > at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded.

    i have a couple of reservations about mbox format. first, a majority
    of users that i need to support have inboxes >10mb and some
    have mailboxes >100mb. thus deleting an older message will require
    rewriting the whole file, and if it's not the last message, will take
    up quite a bit of storage, even on venti.

    second, it's very difficult to cache. for example, suppose i have two
    instances of my mail fs interpreting the same mbox at the same time.
    suppose that i delete the 5th out of 500 messages with the first. the
    second will then have to reread the entire mbox to get its pointers
    right. even if i write an index with the first, the entire index
    needs to be reread.

    third, since large messages tend to be really stinking huge, it is not
    efficient to read entire messages to deduce the mime structure. (ned
    reads the mime structure of each doc on startup.) that last mime part
    may be tens of mb into the

    you'll notice that if one uses email in the way it was used
    traditionally in unix environments, that a 500 message, inbox or 15mb
    message is unreasonable. for traditional uses, mbox format and
    loading the whole stinking thing into memory is a great idea.

    unfortunately, that's now how people use email these days.
    our two machines running upas/fs burn up to 5gb of ram.
    old rewritten mail adds several hundred mb/day to the dump.

    > also file lengths is going to be a problem if i'm going to decode files within the fs.
    >
    > i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs


    i've adapted upas/fs to use a directory for mailboxes (mdir) by default.
    mailbox types that support caching (mdir and imap4) keep their cache
    below Maxcache or Σ of (actively) referenced messages, whichever is bigger.

    i've also adapted the rest of upas, including deliver and marshal to
    understand mdir format. imap4d should be done soon.

    this upas/fs keeps an index, so in most cases, emails are not reread unless they are
    viewed. the index contains some information -- like flags -- not recorded
    in the mail.

    i'd be happy to share a working copy to those interested. but it's
    not ready to be inflicted on the world yet. the "From " line date is
    unresolved and imap4d still thinks it knows how to append a message
    to a mailbox.

    > about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-?


    why not? this seems an arbitrary distinction between the inbox and
    every other mailbox.

    - erik



  9. Re: [9fans] store 9p session-only values using lib9p

    Funny, I've done the same in a different way.
    see mail2fs in contrib/nemo.
    Also, I have some proposal, skip to the end of the mail and let me know
    what you think

    In any case, I'd love to see/try your version of upas/fs et al.

    Instead of adapting upas/fs, I use a mail2fs program that uses
    upas to convert mail into an "unpacked" form. Each mail is a directory.
    A "text" file contains the message text right as you would see it in a mail
    reader (including relative paths for attachments). Each attach is decoded
    and kept in the mail's directory ready to be copied, printed, etc.; if possible,
    using the same file name reported by the attachment.

    I use the very same approach for sending mail. A directory /mail/box/nemo/out
    is spooled to send whichever file is placed on it as a mail. The format for the
    outgoing messages is again similar to what you'd type in a mail writer.

    The whole point is that now you editor (plus couple of scripts) becomes a mail
    reader/writer without understanding anything about mail.

    My proposal:
    Why don't we reach an agreement and start using the very
    same format. I suggest keeping my mbox format but adapting upas/fs to
    understand it, which is a good idea. But I'm open to suggestions.

    thanks

    > From: quanstro@quanstro.net
    > To: 9fans@9fans.net
    > Reply-To: 9fans@9fans.net
    > Date: Thu Jun 12 12:10:32 CET 2008
    > Subject: Re: [9fans] store 9p session-only values using lib9p
    >
    > > at the moment i'm playing only with mbox not imap or pop, i have a version with cache per message that 'works', upas/nedmail and acme/mail are able to read messages 'nicely', but attachments are not decoded.

    >
    > i have a couple of reservations about mbox format. first, a majority
    > of users that i need to support have inboxes >10mb and some
    > have mailboxes >100mb. thus deleting an older message will require
    > rewriting the whole file, and if it's not the last message, will take
    > up quite a bit of storage, even on venti.
    >
    > second, it's very difficult to cache. for example, suppose i have two
    > instances of my mail fs interpreting the same mbox at the same time.
    > suppose that i delete the 5th out of 500 messages with the first. the
    > second will then have to reread the entire mbox to get its pointers
    > right. even if i write an index with the first, the entire index
    > needs to be reread.
    >
    > third, since large messages tend to be really stinking huge, it is not
    > efficient to read entire messages to deduce the mime structure. (ned
    > reads the mime structure of each doc on startup.) that last mime part
    > may be tens of mb into the
    >
    > you'll notice that if one uses email in the way it was used
    > traditionally in unix environments, that a 500 message, inbox or 15mb
    > message is unreasonable. for traditional uses, mbox format and
    > loading the whole stinking thing into memory is a great idea.
    >
    > unfortunately, that's now how people use email these days.
    > our two machines running upas/fs burn up to 5gb of ram.
    > old rewritten mail adds several hundred mb/day to the dump.
    >
    > > also file lengths is going to be a problem if i'm going to decode files within the fs.
    > >
    > > i've put what i have in /n/sources/contrib/gabidiaz/wip/mboxfs

    >
    > i've adapted upas/fs to use a directory for mailboxes (mdir) by default.
    > mailbox types that support caching (mdir and imap4) keep their cache
    > below Maxcache or Σ of (actively) referenced messages, whichever is bigger.
    >
    > i've also adapted the rest of upas, including deliver and marshal to
    > understand mdir format. imap4d should be done soon.
    >
    > this upas/fs keeps an index, so in most cases, emails are not reread unless they are
    > viewed. the index contains some information -- like flags -- not recorded
    > in the mail.
    >
    > i'd be happy to share a working copy to those interested. but it's
    > not ready to be inflicted on the world yet. the "From " line date is
    > unresolved and imap4d still thinks it knows how to append a message
    > to a mailbox.
    >
    > > about the From line, qmail man page about mbox format says it is composed of From space sender@... space current date, and it is generated by the delivery agent, but moving one message from a box to another doesn't use the delivery agent :-?

    >
    > why not? this seems an arbitrary distinction between the inbox and
    > every other mailbox.
    >
    > - erik
    >
    >



  10. Re: [9fans] store 9p session-only values using lib9p

    hello


    great news, i just used mbox to avoid reworking the whole thing at the same time, but you already done it. I was already aware of the mbox-pain.

    i'll be pleased to help on this, test, bug report, try it on a public server or whatever it's needed to finish this up.

    About the date in the From line, it is supposed that the From line is written when the mail is delivered, i mean, the action of saving the email on a file. If you write it in mbox, then no new From will be written when you move that message. I'm not sure i understand the problem here.

    thanks

    gabi


  11. Re: [9fans] store 9p session-only values using lib9p

    I don't understand.
    mail2fs parses attachments. It uses upas/fs, but it parses attachs.


    On Thu, Jun 12, 2008 at 12:39 PM, wrote:
    > Hello
    >
    > I used your mail2fs to store my mail archives for a couple of months (and still use it ☺), but the main problem, which is parse attachments using upas/fs, is not solved.
    >
    > what erik has done is great, i would vote to have it finished soon ☺
    >
    > slds.
    >
    > gabi
    >
    >


  12. Re: [9fans] store 9p session-only values using lib9p

    Hello

    iirc, mail2fs calls upas/fs then copy /mail/fs/box/files to build the new layout, but if you call upas/fs on a message with a 10mb attachment, upas/fs will load the whole thing in ram, mail2fs will write it to a file, then upas/fs will end it s operation.

    but the memory usage will be huge too when parsing. (9grid.es died due to this time ago, the poor thing only has 256mb of ram and it's running venti

    slds.

    gabi


  13. Re: [9fans] store 9p session-only values using lib9p

    It copies *and* parses the attachment, but I admit that you have to
    be able to keep at least the single mail in memory while converting it
    to the new format.

    Should this be a problem, perhaps doing what Russ suggested
    (fixing upas/fs to let it handle big files) would be the right thing.

    In any way, I'd still be converting my mail to the new format, because
    of the convenience of having everything unpacked and ready to read,
    and because of the space it's likely to be saving in venti.

    On Thu, Jun 12, 2008 at 12:55 PM, wrote:
    > Hello
    >
    > iirc, mail2fs calls upas/fs then copy /mail/fs/box/files to build the new layout, but if you call upas/fs on a message with a 10mb attachment, upas/fs will load the whole thing in ram, mail2fs will write it to a file, then upas/fs will end it s operation.
    >
    > but the memory usage will be huge too when parsing. (9grid.es died due to this time ago, the poor thing only has 256mb of ram and it's running venti
    >
    > slds.
    >
    > gabi
    >
    >



  14. Re: [9fans] store 9p session-only values using lib9p

    Hello

    right, that's just one of the problems erik solved and i was still solving ☺, the memory usage of upas/fs.

    slds.

    gabi


  15. Re: [9fans] store 9p session-only values using lib9p

    > About the date in the From line, it is supposed that the From line is written when the mail is delivered, i mean, the action of saving the email on a file. If you write it in mbox, then no new From will be written when you move that message.

    why not? the "From " line is written when the message is delivered to a
    mb. but it's not part of the message proper. so copying a message
    really copies the rfc/822 part. the from line is an awkward appendage.
    it is strange to carry it forward in the special case of copying a mail
    from one box to another.

    - erik



  16. Re: [9fans] store 9p session-only values using lib9p

    > Funny, I've done the same in a different way.
    > see mail2fs in contrib/nemo.
    > Also, I have some proposal, skip to the end of the mail and let me know
    > what you think
    >
    > In any case, I'd love to see/try your version of upas/fs et al.


    /n/sources/contrib/quanstro/src/nupas. cavet emptor.

    nupas/fs is fully compatable with upas/*.

    i have been using the mdir format for about 18 months. but i have
    just recently added the bits to reduce memory footprint.

    i'm currently spending quite a bit of time on this so details may change.

    i would appreciate any feedback.

    > Instead of adapting upas/fs, I use a mail2fs program that uses
    > upas to convert mail into an "unpacked" form. Each mail is a directory.
    > A "text" file contains the message text right as you would see it in a mail
    > reader (including relative paths for attachments). Each attach is decoded
    > and kept in the mail's directory ready to be copied, printed, etc.; if possible,
    > using the same file name reported by the attachment.


    it's hard denying that this is some allure to this idea and i definately
    considered it. however, we get quite a bit of three-part email containing
    the mime parent a 150 byte message and the 400 (with headers) byte
    replyed-to message. depending on the details of your format, this
    could require 3 directories + 3 files or 6 blocks of storage for a
    smaller-than 1 block message. (assuming 8kb blocks.) it also could
    result in quite a bit more seeking when scanning a number of messages.

    in addition, this format is not compatable with any of the existing
    tools. in mdir format, each file looks like a mailbox of exactly one
    message. in addition, in mdir format each message is self-contained.
    i can just cp it.

    finally, a number of email security standards (so-called spf2.0 and
    s/mime) require the original email. it's not clear to me that one can
    reconstruct an original mime message from processed parts without
    leaving some breadcrumbs behind. there's no requirement not to
    base64-encode us-ascii, for example.

    - erik



  17. Re: [9fans] store 9p session-only values using lib9p

    stupid hacks:

    echo 1,10p id from|nupas/nedmail $*>[2=]
    echo 1,10p id flags|nupas/nedmail $*>[2=]

    - erik



  18. Re: [9fans] store 9p session-only values using lib9p

    hello


    hehe, i don't know why it is copied, probably because if you use mbox format you need the From line as a message delimeter, but if you're using other format, the From line could be avoided. As we are talking about email, may be it has a hidden meaning nobody remembers now .

    greetings,

    gabi


  19. Re: [9fans] store 9p session-only values using lib9p

    This mail with a reply inlined would be
    msgdir/
    msgdir/text
    msgdir/raw

    If, the reply was inlined. Anyone wanting to use the original can go use raw.
    But the nice thing is that you can edit text, copy attachments in/out,
    remove them, etc. etc.

    Compatibility in my case is achieved by leaving upas et al. as they
    are; I convert
    just my mail once it entered the system.

    On Thu, Jun 12, 2008 at 2:25 PM, erik quanstrom wrote:
    >> Funny, I've done the same in a different way.
    >> see mail2fs in contrib/nemo.
    >> Also, I have some proposal, skip to the end of the mail and let me know
    >> what you think
    >>
    >> In any case, I'd love to see/try your version of upas/fs et al.

    >
    > /n/sources/contrib/quanstro/src/nupas. cavet emptor.
    >
    > nupas/fs is fully compatable with upas/*.
    >
    > i have been using the mdir format for about 18 months. but i have
    > just recently added the bits to reduce memory footprint.
    >
    > i'm currently spending quite a bit of time on this so details may change.
    >
    > i would appreciate any feedback.
    >
    >> Instead of adapting upas/fs, I use a mail2fs program that uses
    >> upas to convert mail into an "unpacked" form. Each mail is a directory.
    >> A "text" file contains the message text right as you would see it in a mail
    >> reader (including relative paths for attachments). Each attach is decoded
    >> and kept in the mail's directory ready to be copied, printed, etc.; if possible,
    >> using the same file name reported by the attachment.

    >
    > it's hard denying that this is some allure to this idea and i definately
    > considered it. however, we get quite a bit of three-part email containing
    > the mime parent a 150 byte message and the 400 (with headers) byte
    > replyed-to message. depending on the details of your format, this
    > could require 3 directories + 3 files or 6 blocks of storage for a
    > smaller-than 1 block message. (assuming 8kb blocks.) it also could
    > result in quite a bit more seeking when scanning a number of messages.
    >
    > in addition, this format is not compatable with any of the existing
    > tools. in mdir format, each file looks like a mailbox of exactly one
    > message. in addition, in mdir format each message is self-contained.
    > i can just cp it.
    >
    > finally, a number of email security standards (so-called spf2.0 and
    > s/mime) require the original email. it's not clear to me that one can
    > reconstruct an original mime message from processed parts without
    > leaving some breadcrumbs behind. there's no requirement not to
    > base64-encode us-ascii, for example.
    >
    > - erik
    >
    >
    >



  20. Re: [9fans] store 9p session-only values using lib9p

    > why not? the "From " line is written when the message is delivered to a
    > mb. but it's not part of the message proper.


    that may be your mental model, but it's not mine.
    to me, the From line is as much part of the message
    as the rest of the mail. like i said before, it is a
    postmark. it indicates the time the mail was originally
    delivered along with the mail system's idea of the sender.

    there was a time, in the calm pre-rfc822 days
    when the from line was all the header you had.

    for example, here's an entire message locally
    delivered on plan 9 in 1992:

    From ken Tue Sep 8 03:42:43 EDT 1992
    i finally got my copy.

    personally, when i've used mail systems without from lines,
    i've always been annoyed, because then the dates
    on the message are relative to someone else's clock,
    not my local one. i don't mean just time-zone
    variations; plenty of people have their clocks set wrong.
    i like that nedmail shows me the time the message
    arrived, not the claimed date in the header.

    if you view the date that way, as an integral part
    of the delivered message, it would sure be strange
    if saving the message to a different folder altered
    the delivery date. receiving a message and filing
    a message are two very different things.

    > so copying a message
    > really copies the rfc/822 part. the from line is an awkward appendage.
    > it is strange to carry it forward in the special case of copying a mail
    > from one box to another.


    the rfc822 part is the awkward appendage.
    mail used to be simple.

    russ



+ Reply to Thread
Page 1 of 2 1 2 LastLast