Can memory mapping run out of memory after the file's been mapped? - Unix

This is a discussion on Can memory mapping run out of memory after the file's been mapped? - Unix ; If I map a large file as MAP_PRIVATE (ie copy on write) is there any circumstance when writing to the file will cause a memory allocation failure where the OS tries to copy a page and fails? If so how ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: Can memory mapping run out of memory after the file's been mapped?

  1. Can memory mapping run out of memory after the file's been mapped?


    If I map a large file as MAP_PRIVATE (ie copy on write) is there any
    circumstance when writing to the file will cause a memory allocation
    failure where the OS tries to copy a page and fails? If so how would
    it manifest itself and would there be any way of catching this error
    (eg via some signal?) since simply writing to a pointer location
    obviously does not set errno.

    Thanks for any help

    B2003

  2. Re: Can memory mapping run out of memory after the file's beenmapped?

    On Wed, 04 Jun 2008 02:23:21 -0700, thagor2008 wrote:

    > If I map a large file as MAP_PRIVATE (ie copy on write) is there any
    > circumstance when writing to the file will cause a memory allocation
    > failure where the OS tries to copy a page and fails? If so how would it
    > manifest itself and would there be any way of catching this error (eg
    > via some signal?) since simply writing to a pointer location obviously
    > does not set errno.


    On a virtual memory system, the "memory allocation failure" consists of
    the OS not being able to find backing store in a pagefile. It desparately
    needs a disk block to kick out a memory page and cannot allocate one. The
    only way for the OS to survive this situation is to kill some processes,
    freeing their blocks *on the pagefile(s)*.

    Google for oom-handler overcommit

    HTH,
    AvK

  3. Re: Can memory mapping run out of memory after the file's beenmapped?

    On Jun 4, 11:57 am, Moi wrote:
    > On Wed, 04 Jun 2008 02:23:21 -0700, thagor2008 wrote:
    > > If I map a large file as MAP_PRIVATE (ie copy on write) is there any
    > > circumstance when writing to the file will cause a memory allocation
    > > failure where the OS tries to copy a page and fails? If so how would it
    > > manifest itself and would there be any way of catching this error (eg
    > > via some signal?) since simply writing to a pointer location obviously
    > > does not set errno.

    >
    > On a virtual memory system, the "memory allocation failure" consists of
    > the OS not being able to find backing store in a pagefile. It desparately
    > needs a disk block to kick out a memory page and cannot allocate one. The
    > only way for the OS to survive this situation is to kill some processes,
    > freeing their blocks *on the pagefile(s)*.


    Yes , thanks , I'm aware of how OS's work in that respect but it
    doesn't answer my question. WIll this ever happen with memory mapping
    (I'm guessing yes it could) and if so is there any way for my app to
    find out about it in the same way that malloc() will return a NULL?

    B2003


  4. Re: Can memory mapping run out of memory after the file's beenmapped?

    On Wed, 04 Jun 2008 04:19:49 -0700, thagor2008 wrote:

    > On Jun 4, 11:57 am, Moi wrote:
    >> On Wed, 04 Jun 2008 02:23:21 -0700, thagor2008 wrote:
    >> > If I map a large file as MAP_PRIVATE (ie copy on write) is there any
    >> > circumstance when writing to the file will cause a memory allocation
    >> > failure where the OS tries to copy a page and fails? If so how would
    >> > it manifest itself and would there be any way of catching this error
    >> > (eg via some signal?) since simply writing to a pointer location
    >> > obviously does not set errno.

    >>
    >> On a virtual memory system, the "memory allocation failure" consists of
    >> the OS not being able to find backing store in a pagefile. It
    >> desparately needs a disk block to kick out a memory page and cannot
    >> allocate one. The only way for the OS to survive this situation is to
    >> kill some processes, freeing their blocks *on the pagefile(s)*.

    >
    > Yes , thanks , I'm aware of how OS's work in that respect but it doesn't
    > answer my question. WIll this ever happen with memory mapping (I'm
    > guessing yes it could) and if so is there any way for my app to find out
    > about it in the same way that malloc() will return a NULL?
    >


    Your program will never find out, because it is not runnable, but waiting
    for the page to be present.
    *after* the page is present the program could of course read the clock and
    calculate how log it took.

    See also getrusage(2), which reports the number of pagefaults (and lots of
    other stuff).

    HTH,
    AvK

  5. Re: Can memory mapping run out of memory after the file's beenmapped?

    Moi schrieb:
    > On Wed, 04 Jun 2008 02:23:21 -0700, thagor2008 wrote:
    >
    >> If I map a large file as MAP_PRIVATE (ie copy on write) is there any
    >> circumstance when writing to the file will cause a memory allocation
    >> failure where the OS tries to copy a page and fails? If so how would it
    >> manifest itself and would there be any way of catching this error (eg
    >> via some signal?) since simply writing to a pointer location obviously
    >> does not set errno.

    >
    > On a virtual memory system, the "memory allocation failure" consists of
    > the OS not being able to find backing store in a pagefile. It desparately
    > needs a disk block to kick out a memory page and cannot allocate one. The
    > only way for the OS to survive this situation is to kill some processes,
    > freeing their blocks *on the pagefile(s)*.
    >
    > Google for oom-handler overcommit
    >
    > HTH,
    > AvK


    I think that is not quite correct:
    SUS says:
    An implementation may generate SIGBUS signals when a reference would
    cause an error in the mapped object, such as out-of-space condition.

    So you really have to look into the docs of your OS. Solaris, for
    example has an option to mmap without reservation (MAP_NORESERVE), and
    will do exactly this when the mmap'ed object's backing store runs out of
    memory.

    - Thomas

  6. Re: Can memory mapping run out of memory after the file's beenmapped?

    On Wed, 04 Jun 2008 13:40:04 +0200, Thomas Maier-Komor wrote:

    > Moi schrieb:


    >> On a virtual memory system, the "memory allocation failure" consists of
    >> the OS not being able to find backing store in a pagefile. It
    >> desparately needs a disk block to kick out a memory page and cannot
    >> allocate one. The only way for the OS to survive this situation is to
    >> kill some processes, freeing their blocks *on the pagefile(s)*.
    >>
    >> Google for oom-handler overcommit
    >>
    >> HTH,
    >> AvK

    >
    > I think that is not quite correct:
    > SUS says:
    > An implementation may generate SIGBUS signals when a reference would
    > cause an error in the mapped object, such as out-of-space condition.
    >
    > So you really have to look into the docs of your OS. Solaris, for
    > example has an option to mmap without reservation (MAP_NORESERVE), and
    > will do exactly this when the mmap'ed object's backing store runs out of
    > memory.


    I stand corrected. Linux has MAP_NORESERVE, too, but sends a SIGSEGV on
    failure. But, from an application point of view it will probably be hard
    to deal with these signals in a useful way ;-[

    AvK

  7. Re: Can memory mapping run out of memory after the file's beenmapped?

    Moi schrieb:
    > On Wed, 04 Jun 2008 13:40:04 +0200, Thomas Maier-Komor wrote:
    >
    > I stand corrected. Linux has MAP_NORESERVE, too, but sends a SIGSEGV on
    > failure. But, from an application point of view it will probably be hard
    > to deal with these signals in a useful way ;-[
    >
    > AvK


    In stating that your comment is not quite correct, I meant to say that
    it is incomplete and that there are more situations that can lead to an
    issue that ought to be considered.
    BTW: are you sure that Linux delivers SIGSEGV? That would be a violation
    of the SUS...

    - Thomas

  8. Re: Can memory mapping run out of memory after the file's beenmapped?

    On Wed, 04 Jun 2008 14:01:18 +0200, Thomas Maier-Komor wrote:

    >
    > BTW: are you sure that Linux delivers SIGSEGV? That would be a violation
    > of the SUS...


    Well, I just checked the manpage. Personally I never used MAP_PRIVATE|
    MAP_NORESERVE.
    Would be pretty advanced to construct some (sigjump?) scheme to deal with
    these page-not-present conditions...

    AvK

+ Reply to Thread