file access via read/write vs mmap() interface - Unix

This is a discussion on file access via read/write vs mmap() interface - Unix ; Hi Everyone, Over the last one week, i was trying few programs with mmap() function and i have a bit of understanding of its use and purpose. As per my current understanding, An application accessing a file with a typical ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: file access via read/write vs mmap() interface

  1. file access via read/write vs mmap() interface

    Hi Everyone,

    Over the last one week, i was trying few programs with mmap()
    function and i have a bit of understanding of its use and purpose. As
    per my current understanding,

    An application accessing a file with a typical read/write interface
    doesn't need to worry about the size of the file that it can support
    and it can support files of any size.

    However, an application accessing a file with a mmap() approach,
    can't access a file of any size, as it is based on virtual memory
    subsystem, i can address only a limited number of address space.
    [example, 32 bit micro-processor with 2 GB RAM and 2 GB Virtual
    Memory] can't support any file access with mmap() whose size is more
    than 4 GB.

    Is my understanding correct? If not, please explain.

    Thanks in advance!!!


  2. Re: file access via read/write vs mmap() interface

    sam_cit@yahoo.co.in writes:
    > An application accessing a file with a typical read/write interface
    > doesn't need to worry about the size of the file that it can support
    > and it can support files of any size.
    >
    > However, an application accessing a file with a mmap() approach,
    > can't access a file of any size, as it is based on virtual memory
    > subsystem, i can address only a limited number of address space.
    > [example, 32 bit micro-processor with 2 GB RAM and 2 GB Virtual
    > Memory] can't support any file access with mmap() whose size is more
    > than 4 GB.
    >
    > Is my understanding correct? If not, please explain.


    An application using read/write can't access a file of an arbitrary
    size because it cannot allocate an arbitrarily large buffer to read it
    into. An application using mmap, on the other hand, can access a file
    of any size, because it doesn't need to read all of the contents of it
    into a single buffer, but can map each part of the file seperately,
    after having unmapped the previously mapped part.

    That's your statement, just reversed (and as wrong). Moreover, all of
    this is painfully explicit just from reading the documentation alone.


  3. Re: file access via read/write vs mmap() interface

    sam_cit@yahoo.co.in wrote:
    > Hi Everyone,
    >
    > Over the last one week, i was trying few programs with mmap()
    > function and i have a bit of understanding of its use and purpose. As
    > per my current understanding,
    >
    > An application accessing a file with a typical read/write interface
    > doesn't need to worry about the size of the file that it can support
    > and it can support files of any size.
    >
    > However, an application accessing a file with a mmap() approach,
    > can't access a file of any size, as it is based on virtual memory
    > subsystem, i can address only a limited number of address space.
    > [example, 32 bit micro-processor with 2 GB RAM and 2 GB Virtual
    > Memory] can't support any file access with mmap() whose size is more
    > than 4 GB.
    >
    > Is my understanding correct? If not, please explain.
    >
    > Thanks in advance!!!
    >


    You can map the components of the file you want into memory.

    Also, you can get a 64 bit machine giving you plenty of addresses.

    Using the read/write interface accomplishes the same thing as mmap (to a
    first approximation). The advantage is that mmap bypasses the file
    buffers and avoids making multiple copies of the content. The
    read/write solution to this is to use "O_DIRECT" which can perform just
    as fast as the VM system, but then, for IPC, using a shared mapped file
    is about as fast as you're ever going to get.

  4. Re: file access via read/write vs mmap() interface

    On Apr 25, 6:43 pm, Gianni Mariani wrote:
    > sam_...@yahoo.co.in wrote:
    > > Hi Everyone,

    >
    > > Over the last one week, i was trying few programs with mmap()
    > > function and i have a bit of understanding of its use and purpose. As
    > > per my current understanding,

    >
    > > An application accessing a file with a typical read/write interface
    > > doesn't need to worry about the size of the file that it can support
    > > and it can support files of any size.

    >
    > > However, an application accessing a file with a mmap() approach,
    > > can't access a file of any size, as it is based on virtual memory
    > > subsystem, i can address only a limited number of address space.
    > > [example, 32 bit micro-processor with 2 GB RAM and 2 GB Virtual
    > > Memory] can't support any file access with mmap() whose size is more
    > > than 4 GB.

    >
    > > Is my understanding correct? If not, please explain.

    >
    > > Thanks in advance!!!

    >
    > You can map the components of the file you want into memory.
    >
    > Also, you can get a 64 bit machine giving you plenty of addresses.
    >
    > Using the read/write interface accomplishes the same thing as mmap (to a
    > first approximation). The advantage is that mmap bypasses the file
    > buffers and avoids making multiple copies of the content. The
    > read/write solution to this is to use "O_DIRECT" which can perform just
    > as fast as the VM system, but then, for IPC, using a shared mapped file
    > is about as fast as you're ever going to get.- Hide quoted text -
    >
    > - Show quoted text -



    Yes i understand the limit of address space in 32 bit machine while
    using mmap() to access a file.
    My doubt is that as to whether such a limit holds true for a file
    access via read/write interface?


  5. Re: file access via read/write vs mmap() interface

    sam_cit@yahoo.co.in writes:
    > On Apr 25, 6:43 pm, Gianni Mariani wrote:
    >> sam_...@yahoo.co.in wrote:
    >> > Over the last one week, i was trying few programs with mmap()
    >> > function and i have a bit of understanding of its use and purpose. As
    >> > per my current understanding,

    >>
    >> > An application accessing a file with a typical read/write interface
    >> > doesn't need to worry about the size of the file that it can support
    >> > and it can support files of any size.

    >>
    >> > However, an application accessing a file with a mmap() approach,
    >> > can't access a file of any size, as it is based on virtual memory
    >> > subsystem, i can address only a limited number of address space.
    >> > [example, 32 bit micro-processor with 2 GB RAM and 2 GB Virtual
    >> > Memory] can't support any file access with mmap() whose size is more
    >> > than 4 GB.

    >>
    >> > Is my understanding correct? If not, please explain.

    >>
    >> > Thanks in advance!!!

    >>
    >> You can map the components of the file you want into memory.
    >>
    >> Also, you can get a 64 bit machine giving you plenty of addresses.
    >>
    >> Using the read/write interface accomplishes the same thing as mmap (to a
    >> first approximation). The advantage is that mmap bypasses the file
    >> buffers and avoids making multiple copies of the content. The
    >> read/write solution to this is to use "O_DIRECT" which can perform just
    >> as fast as the VM system, but then, for IPC, using a shared mapped file
    >> is about as fast as you're ever going to get.- Hide quoted text -
    >>
    >> - Show quoted text -

    >
    >
    > Yes i understand the limit of address space in 32 bit machine while
    > using mmap() to access a file.
    > My doubt is that as to whether such a limit holds true for a file
    > access via read/write interface?


    Sorry to be so harsh, but, taking your performance up to now into
    account, you understand about nothing regarding the topics you keep
    posting trash about and appear to be way to stubborn to ever learn.
    So, please, call it a day and look for a greener pasture elswhere.

  6. Re: file access via read/write vs mmap() interface

    >
    > An application using read/write can't access a file of an arbitrary
    > size because it cannot allocate an arbitrarily large buffer to read it
    > into.
    >


    Sorry if you didn't get the correct picture of the application. I'm
    not talking about the application which allocates a buffer and reads
    file contents into the buffer.
    It is something like

    for ( till end of file)
    keep reading the file one character at a time using fread().

    Now, i'm not allocating any memory at the application layer and the
    operating system takes care of the buffer management of the file.
    Hence i would assume that there shouldn't be any limit on file size
    for such applications.






  7. Re: file access via read/write vs mmap() interface

    On Wed, 25 Apr 2007 00:23:42 -0700, sam_cit wrote:

    > However, an application accessing a file with a mmap() approach,
    > can't access a file of any size, as it is based on virtual memory
    > subsystem, i can address only a limited number of address space.
    > [example, 32 bit micro-processor with 2 GB RAM and 2 GB Virtual
    > Memory] can't support any file access with mmap() whose size is more
    > than 4 GB.
    >
    > Is my understanding correct? If not, please explain.


    Nope: When you mmap a file, you are not
    using any virtual memory and you are not using any RAM,
    you are only using address space and disk.

    When you access the mapped addresses, RAM from the page
    pool is allocated and the disk page loaded into the RAM.

    Your mmap()ed file acts exactly like a swap partition,
    except that

    * only you can access it, the OS doesn't use it

    * swap partition is specially formatted for performance
    wheres an ordinary disk file could be scattered all over
    the disk due to fragmentation etc.

    So in fact the binding:

    virtual address --> physical RAM address

    is NOT created by mmap: it creates a binding:

    virtual address --> disk address

    It is only your access to the virtual address
    which forces the OS to allocate a physical RAM address
    to your virtual address. This is typically done by
    marking the page 'inaccessible' to force the processor
    to generate a page fault interrupt, and then the OS
    takes over and tries to find a free RAM page to plug in,
    then after that it schedules disk I/O and then puts your
    process to sleep until the I/O is done.

    When you wake up the disk block is in memory.. and you have:

    virtual address --> RAM address --> disk address

    So an accessible mapping actually has THREE addresses.

    --
    John Skaller
    Felix, successor to C++: http://felix.sf.net

  8. Re: file access via read/write vs mmap() interface

    In article <1177595804.799900.110790@t39g2000prd.googlegroups. com>,
    sam_cit@yahoo.co.in wrote:

    > >
    > > An application using read/write can't access a file of an arbitrary
    > > size because it cannot allocate an arbitrarily large buffer to read it
    > > into.
    > >

    >
    > Sorry if you didn't get the correct picture of the application. I'm
    > not talking about the application which allocates a buffer and reads
    > file contents into the buffer.
    > It is something like
    >
    > for ( till end of file)
    > keep reading the file one character at a time using fread().
    >
    > Now, i'm not allocating any memory at the application layer and the
    > operating system takes care of the buffer management of the file.
    > Hence i would assume that there shouldn't be any limit on file size
    > for such applications.


    His point was that just like you don't have to read the entire file at
    once, you don't have to mmap the entire file all at once, either.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

+ Reply to Thread