malloc more than physical memory - Linux

This is a discussion on malloc more than physical memory - Linux ; I'm trying to allocate an array that is larger than my physical memory size. My shell limit is set to unlimited, and I've tested this on two identical Linux 2.6.9 Opteron machines that have 4GB of DRAM and 8GB of ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: malloc more than physical memory

  1. malloc more than physical memory

    I'm trying to allocate an array that is larger than my physical memory
    size. My shell limit is set to unlimited, and I've tested this on two
    identical Linux 2.6.9 Opteron machines that have 4GB of DRAM and 8GB of
    memory respectively. On the latter machine, malloc indeed succeeds.
    However this fails on the identical 4GB machine. My test program is
    below. How can I get this to succeed on the 4GB Opteron? The working
    set should be fairly local so paging to disk should work fine.

    int main() {

    int* f;
    f = malloc(8589934592);

    if (f != 0) {
    printf("malloced\n");
    } else {
    printf("NO malloced\n");
    }


    while (1) ;
    return 0;
    }


  2. Re: malloc more than physical memory


    ppg wrote:
    > I'm trying to allocate an array that is larger than my physical memory
    > size. My shell limit is set to unlimited, and I've tested this on two
    > identical Linux 2.6.9 Opteron machines that have 4GB of DRAM and 8GB of
    > memory respectively. On the latter machine, malloc indeed succeeds.
    > However this fails on the identical 4GB machine. My test program is
    > below. How can I get this to succeed on the 4GB Opteron? The working
    > set should be fairly local so paging to disk should work fine.
    >
    > int main() {
    >
    > int* f;
    > f = malloc(8589934592);
    >
    > if (f != 0) {
    > printf("malloced\n");
    > } else {
    > printf("NO malloced\n");
    > }
    >
    >
    > while (1) ;
    > return 0;
    > }


    How much swap space do the two machine have? And how is memory
    overcommittment set?

    DS


  3. Re: malloc more than physical memory

    David Schwartz wrote:

    >
    > ppg wrote:
    >> I'm trying to allocate an array that is larger than my physical memory
    >> size. My shell limit is set to unlimited, and I've tested this on two
    >> identical Linux 2.6.9 Opteron machines that have 4GB of DRAM and 8GB of
    >> memory respectively. On the latter machine, malloc indeed succeeds.
    >> However this fails on the identical 4GB machine. My test program is
    >> below. How can I get this to succeed on the 4GB Opteron? The working
    >> set should be fairly local so paging to disk should work fine.
    >>
    >> int main() {
    >>
    >> int* f;
    >> f = malloc(8589934592);
    >>
    >> if (f != 0) {
    >> printf("malloced\n");
    >> } else {
    >> printf("NO malloced\n");
    >> }
    >>
    >>
    >> while (1) ;
    >> return 0;
    >> }

    >
    > How much swap space do the two machine have? And how is memory
    > overcommittment set?

    And what setting of ulimit are used ?

  4. Re: malloc more than physical memory


    >
    > How much swap space do the two machine have? And how is memory
    > overcommittment set?
    >


    Thank you. The swap file is the problem as it is sized to only 2GB.


  5. Re: malloc more than physical memory

    But how will it matter? The memory allocated will be virtual memory.
    That will be in virtual memory space of process.
    David Schwartz wrote:
    > ppg wrote:
    > > I'm trying to allocate an array that is larger than my physical memory
    > > size. My shell limit is set to unlimited, and I've tested this on two
    > > identical Linux 2.6.9 Opteron machines that have 4GB of DRAM and 8GB of
    > > memory respectively. On the latter machine, malloc indeed succeeds.
    > > However this fails on the identical 4GB machine. My test program is
    > > below. How can I get this to succeed on the 4GB Opteron? The working
    > > set should be fairly local so paging to disk should work fine.
    > >
    > > int main() {
    > >
    > > int* f;
    > > f = malloc(8589934592);
    > >
    > > if (f != 0) {
    > > printf("malloced\n");
    > > } else {
    > > printf("NO malloced\n");
    > > }
    > >
    > >
    > > while (1) ;
    > > return 0;
    > > }

    >
    > How much swap space do the two machine have? And how is memory
    > overcommittment set?
    >
    > DS



  6. Re: malloc more than physical memory

    "miline" writes:

    > But how will it matter? The memory allocated will be virtual memory.
    > That will be in virtual memory space of process.


    To be used, there has to be a place in swap space to back the VM up
    to. So, the total space used by all processes normally can't exceed
    the system's swap space -- if you try a malloc() that'll overflow
    that, it will fail.

    As many processes allocate a lot more memory than they ever use, it's
    possible to turn on overcommit. When you do this the malloc() will
    succeed regardless of whether there's sufficient swap space; it won't
    try to find space to back up the VM until it's actually accessed, and
    won't fail until then.
    --
    Joseph J. Pfeiffer, Jr., Ph.D. Phone -- (505) 646-1605
    Department of Computer Science FAX -- (505) 646-1002
    New Mexico State University http://www.cs.nmsu.edu/~pfeiffer

  7. Re: malloc more than physical memory


    miline wrote:
    > But how will it matter? The memory allocated will be virtual memory.
    > That will be in virtual memory space of process.


    Unless the system is set to overcommit, it will not allow you to
    allocate more virtual memory than it can back. Otherwise, if you used
    all the virtual memory, the machine would run out of swap and have to
    kill a process.

    If overcommittment is off, the system makes every attempt to make calls
    to things like 'malloc' and 'fork' fail if the alternative might be
    that the system must kill a process just because it dirtied a page it
    thought it already allocated.

    DS


+ Reply to Thread