Behavior of malloc/free on AIX - Aix

This is a discussion on Behavior of malloc/free on AIX - Aix ; Hello AIX experts, I am investigating a possible memory leak issue in our application on AIX platform. The same application works fine on other unix flavors (Solaris). To reproduce this problem in our environment, I wrote a sample test code ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: Behavior of malloc/free on AIX

  1. Behavior of malloc/free on AIX

    Hello AIX experts,

    I am investigating a possible memory leak issue in our application on
    AIX platform. The same application works fine on other unix flavors
    (Solaris). To reproduce this problem in our environment, I wrote a
    sample test code (see below for more details) to observe the memory
    management/usage. Flow is :
    - Allocate memory using malloc
    - take ps gv output of the process
    - free the allocated memory
    - take final ps gv output of the process.

    I expected that ps gv output will show reduced memory usage after
    calling free. To my surprise, it didn't.

    Can you please explain the reason for this?

    Also, I would appreciate if you can throw some light on AIX memory
    management and any pointers on how to troubleshoot these kind of
    issues on AIX platform.

    For your reference, I have shared code here:

    #include
    #include
    #include
    int main ()
    {
    char sbuff[1000];
    sprintf(sbuff,"echo \"Before malloc function \" >>
    testoutput2.txt");
    system(sbuff);
    sprintf(sbuff,"date >> testoutput2.txt");
    system(sbuff);
    pid_t pid = getpid();
    sprintf(sbuff,"ps gv %d >> testoutput2.txt",pid);
    system(sbuff);

    int * buffer1, * buffer2;
    buffer1 = (int*) malloc (10000 * sizeof(int));
    buffer2 = (int*) malloc (20000 * sizeof(int));

    sprintf(sbuff,"echo \"After malloc function \" >>
    testoutput2.txt");
    system(sbuff);
    sprintf(sbuff,"date >> testoutput2.txt");
    system(sbuff);
    sprintf(sbuff,"ps gv %d >> testoutput2.txt",pid);
    system(sbuff);

    free (buffer1);
    free (buffer2);

    sprintf(sbuff,"echo \"After free function \" >>
    testoutput2.txt");
    system(sbuff);
    sprintf(sbuff,"date >> testoutput2.txt");
    system(sbuff);
    sprintf(sbuff,"ps gv %d >> testoutput2.txt",pid);
    system(sbuff);

    return 0;
    }

    The output of the above sample program is:

    Before malloc function
    Mon Oct 15 19:24:20 IST 2007
    PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM
    COMMAND
    47114 pts/8 A 0:00 0 176 200 32768 9 24 0.0
    0.0 ./test
    After malloc function
    Mon Oct 15 19:24:20 IST 2007
    PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM
    COMMAND
    47114 pts/8 A 0:00 0 256 280 32768 9 24 0.0
    0.0 ./test
    After free function
    Mon Oct 15 19:24:20 IST 2007
    PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM
    COMMAND
    47114 pts/8 A 0:00 0 256 280 32768 9 24 0.0
    0.0 ./test


    Thanks in advance
    Shashanka


  2. Re: Behavior of malloc/free on AIX

    shashanka arnady writes:

    > I expected that ps gv output will show reduced memory usage after
    > calling free. To my surprise, it didn't.


    Your expectation is completely wrong.

    You will not see such reduction on any UNIX system (when using
    system libc), except possibly Linux.

    > Can you please explain the reason for this?


    No "standard" malloc implementation (except the one in glibc)
    releases memory obtained by sbrk() back to the system. Instead,
    that memory is kept to satisfy future malloc requests without
    executing any system calls.

    > Also, I would appreciate if you can throw some light on AIX memory
    > management and any pointers on how to troubleshoot these kind of
    > issues on AIX platform.


    To troubleshoot memory leak, use any of the fine heap debuggers
    available. You can start with freeware (dmalloc, ccmalloc, mpatrol),
    but they may not work correctly unless you relink libc.a(shr.o)
    with rtl_enable. Some of the commercial tools you could use are
    Insure++, Purify and ZeroFault.

    Cheers,
    --
    In order to understand recursion you must first understand recursion.
    Remove /-nsp/ for email.

  3. Re: Behavior of malloc/free on AIX

    shashanka arnady schrieb:

    >
    > Also, I would appreciate if you can throw some light on AIX memory
    > management and any pointers on how to troubleshoot these kind of
    > issues on AIX platform.
    >


    get
    http://publib.boulder.ibm.com/infoce...c/genprogc.pdf
    and read chapter 19.

    On your 'memlory leak' see

    Malloc Disclaim
    Malloc Disclaim is an optional extension of the malloc subsystem,
    providing the user with a means to enable the automatic disclaiming of
    memory returned by the free subroutine. This is useful in instances
    where a process has a high paging-space usage, but is not actually using
    the memory. Malloc Disclaim is not enabled by default. It can be enabled
    and configured prior to process startup through the MALLOCOPTIONS
    environment variable, as follows: MALLOCOPTIONS=disclaim

  4. Re: Behavior of malloc/free on AIX

    Hi Paul/Thomas,

    Thanks for your suggestions.

    To disclaim the memory after free system call, I tried using the below
    system calls:
    (as per the
    http://publib.boulder.ibm.com/infoce...rf1/malloc.htm)

    mallopt(M_MXFAST, 0);
    mallopt(M_DISCLAIM, 0);

    There is no decrease in memory shown by 'ps gv' command.

    Before malloc function
    Wed Oct 17 14:42:22 IST 2007
    PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM
    COMMAND
    29700 pts/5 A 0:00 0 176 200 32768 9 24 0.0
    0.0 ./test2

    After malloc function
    Wed Oct 17 14:42:22 IST 2007
    PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM
    COMMAND
    29700 pts/5 A 0:00 0 256 280 32768 9 24 0.0
    0.0 ./test2

    After free function
    Wed Oct 17 14:42:23 IST 2007
    PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM
    COMMAND
    29700 pts/5 A 0:00 0 256 280 32768 9 24 0.0
    0.0 ./test2


    I got following extract from some other IBM help documentation:

    "Specifies a mallopt subroutine command. If M_DISCLAIM is used, then
    the paging space and physical memory in use by freed malloc space is
    returned to the system resource pool. If they are needed to fulfill a
    malloc request, they will be allocated to the process as needed. The
    address space is not altered. This will only release whole pages at a
    time."

    And, following extract from http://coding.derkeiler.com/Archive/...4-06/0856.html

    "most AIX programs can trivially return some memory to the system
    using the mallopt function with the M_DISCLAIM command, which tells
    the malloc subsystem to return freed pages to the OS (via the disclaim
    system call). This is completely transparent to the application - they
    remain mapped in the app's address space, and will be reallocated from
    the VMM pool if and when they're needed to satisfy future malloc
    requests. "

    Does this mean, there is no way to reflect the free'd memory in 'ps
    gv' command output? Please comment. I even tried with 3.1 memory
    allocation policy for the process by setting mallopt(M_MXFAST, 1);

    We actually tried to use Purify for memory leak, but we didn't get any
    clues. Using dmalloc is little difficult as we need to download source
    and configure the binaries. But, we are planning to explore dmalloc/
    ccmalloc/mpatrol. You have any suggestion on which one is best among
    them?

    Thanks in advance.

    Regards
    Hassan


    On Oct 16, 9:05 am, Thomas Braunbeck
    wrote:
    > shashanka arnady schrieb:
    >
    >
    >
    > > Also, I would appreciate if you can throw some light on AIX memory
    > > management and any pointers on how to troubleshoot these kind of
    > > issues on AIX platform.

    >
    > gethttp://publib.boulder.ibm.com/infocenter/pseries/v5r3/topic/com.ibm.a...
    > and read chapter 19.
    >
    > On your 'memlory leak' see
    >
    > Malloc Disclaim
    > Malloc Disclaim is an optional extension of the malloc subsystem,
    > providing the user with a means to enable the automatic disclaiming of
    > memory returned by the free subroutine. This is useful in instances
    > where a process has a high paging-space usage, but is not actually using
    > the memory. Malloc Disclaim is not enabled by default. It can be enabled
    > and configured prior to process startup through the MALLOCOPTIONS
    > environment variable, as follows: MALLOCOPTIONS=disclaim




  5. Re: Behavior of malloc/free on AIX

    I've had the "opportunity" to study these effects on AIX quite
    frequently.

    "DISCLAIM" ... is an extremely expensive operation, and I would not
    recommend it
    for production use. If you have a long running application that does
    not leak
    memory, eventually it will reach stabilization without using disclaim.

    "ps" ... is inaccurate for doing detailed studies of memory usage.
    It is designed to give you a quick view of memory usage so
    that it runs very quickly.

    "svmon" ... is the AIX utility of choice when it comes to analyzing
    heap usage.
    The problem: it only runs as root.

    Solution: change the permissions: chmod 6555 /usr/sbin/svmon
    (i.e., I just don't understand the perceived security exposures IBM
    does
    in protecting svmon from normal users. Abused, of course it can put
    a lot of needless activity on the system ... so can "sar" ).

    General Observations:
    1. malloc runs by acquiring system storage via sbrk.
    Once you have "dirtied" a page, it is part of your working set.
    DISCLAIM can reverse those effects ... at a substantial cost.

    2. So unused memory is part of your working set ... who cares?
    If the system needs memory your free'd (unused) heap is holding,
    it will page them out. There is no further performance effect.

    3. If the application has dynamic memory usage, then those free
    pages will get reused (re-touched, re-dirtied) when they are used
    in subsequent malloc requests.
    Eventually, a well-behaved application will stabilize it's memory
    footprint on the system. That is what you look for.
    We have applications that take as long as 2-12 hours to stabilize
    on the active memory footprint (dirty pages).

    4. svmon can provide very detailed insight into actual memory usage,
    especially across multi-process apps that use shared memory.
    One huge malloc/free can permanently move the sbrk line of
    system allocated memory which gives the perception the app
    is using a lot of memory. It may not be, and as the app
    stabilizes
    that free'd storage gets reused or swapped out.

    5. The primary "take away" from the "vmstat" command is an answer to
    the
    question: Are you paging?
    If the answer (taken from the middle columns of the vmstat output)
    is "yes",
    then your application may be mis-configured ... allocating more
    real memory to cache your results than you have physical memory.
    That is something to worry about ... and not the fact that the app
    has a large (residual) memory footprint.


+ Reply to Thread