Need to know if my application is leaking or not - Unix

This is a discussion on Need to know if my application is leaking or not - Unix ; Using "pmap -x | grep heap" command line to monitor the allocated memory in the heap by a program, I saw an increase of this memory by 4MB from time to time. At startup, I have: 000000010028A000 24 24 16 ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: Need to know if my application is leaking or not

  1. Need to know if my application is leaking or not

    Using "pmap -x | grep heap" command line to monitor the
    allocated memory in the heap by a program, I saw an increase of this
    memory by 4MB from time to time.

    At startup, I have:

    000000010028A000 24 24 16 - rwx--
    [ heap ]
    0000000100290000 1472 1408 512 - rwx--
    [ heap ]
    0000000100400000 36864 36864 36864 - rwx--
    [ heap ]


    The 3rd line is about 4MB.

    Then after a certain time, I have:

    000000010028A000 24 24 16 - rwx--
    [ heap ]
    0000000100290000 1472 1408 512 - rwx--
    [ heap ]
    0000000100400000 40960 40960 40960 - rwx--
    [ heap ]

    So, got an increase by 4MB

    Then after some time, I got:
    000000010028A000 24 24 16 - rwx--
    [ heap ]
    0000000100290000 1472 1408 512 - rwx--
    [ heap ]
    0000000100400000 45056 45056 45056 - rwx--
    [ heap ]

    So another increase by 4MB.

    At the end of the test (data processed), the heap in use is:
    000000010028A000 24 24 16 - rwx--
    [ heap ]
    0000000100290000 1472 1408 512 - rwx--
    [ heap ]
    0000000100400000 73728 73728 73728 - rwx--
    [ heap ]

    Another increase by (7 * 4MB).

    Running then a purified version of the program (a huge one) with the
    same data, purify does not report any leak.

    So, How to interpret this increase of the heap reported by the pmap ?
    Is this a leak even though purify does not report any MLK entry.

    Thanks.


  2. Re: Need to know if my application is leaking or not

    abd_web writes:

    > At the end of the test (data processed), the heap in use is:

    ....
    > Another increase by (7 * 4MB).


    Double your test data size, then double it again.

    If the heap keeps increasing, you have a leak, or a data structure
    that keeps growing.

    > Running then a purified version of the program (a huge one) with the
    > same data, purify does not report any leak.


    The following program will show no leak, yet it has a problem
    (which will cause it to die with memory exhaustion if it is run
    over large enough input file).

    vector v;
    while (fgets(buf, sizeof(buf), fp)) {
    v.push_back(buf); // grows without bounds
    // process current line
    }

    > So, How to interpret this increase of the heap reported by the pmap ?
    > Is this a leak even though purify does not report any MLK entry.


    It may not be a leak, but you may still have a problem.

    Cheers,
    --
    In order to understand recursion you must first understand recursion.
    Remove /-nsp/ for email.

  3. Re: Need to know if my application is leaking or not

    Maybe you could try the `valgrind' tool for checking leaks, which is
    very easy to use.

    --
    William

  4. Re: Need to know if my application is leaking or not

    William Xu wrote:

    > Maybe you could try the `valgrind' tool for checking leaks, which is
    > very easy to use.


    He used Purify, which is functionally similar.
    http://en.wikipedia.org/wiki/IBM_Rational_Purify

  5. Re: Need to know if my application is leaking or not

    On 2 nov, 11:59, Spoon wrote:
    > William Xu wrote:
    > > Maybe you could try the `valgrind' tool for checking leaks, which is
    > > very easy to use.

    >
    > He used Purify, which is functionally similar.http://en.wikipedia.org/wiki/IBM_Rational_Purify


    If you build and run the following program which is BTW not leaking
    memory at all and monitor the heap using 'pmap | grep heap':

    #include
    #include
    #include

    int main ()
    {
    char *buffer1;
    char *buffer2;
    char *buffer3;
    int size;
    char* str =
    "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxx";
    size = strlen(str);

    printf ("\nSleeping 10 sec before 1st allocation\n");
    sleep(10);
    buffer1 = (char*) malloc (1024*1024*4*sizeof(char));
    buffer1 = strncpy(buffer1, str, size);
    printf ("Sleeping 10 sec after 1st allocation\n");
    sleep(10);

    free (buffer1);
    printf ("Sleeping 10 sec after 1st free\n");
    sleep(10);

    buffer2 = (char*) malloc (1024*1024*8*sizeof(char));
    strncpy(buffer2, str, size);

    printf ("Sleeping 10 sec after 2nd allocation\n");
    sleep(10);

    free (buffer2);
    printf ("Sleeping 10 sec after 2nd free\n");
    sleep(10);

    buffer3 = (char*) malloc (1024*1024*16*sizeof(char));
    strncpy(buffer3, str, size);

    printf ("Sleeping 10 sec after 3rd allocation\n");
    sleep(10);

    free (buffer3);
    printf ("Sleeping 1000 sec after 3rd free\n");
    sleep(1000);


    printf ("END\n");
    return 0;
    }

    you will see something like:

    Before any malloc()
    RSS
    1032
    0000000100102000 56K rwx-- [ heap ]
    --------------------------

    After 1st malloc():
    RSS
    5128
    0000000100102000 3064K rwx-- [ heap ]
    0000000100400000 4096K rwx-- [ heap ]
    --------------------------

    After 1st free():
    RSS
    5128
    0000000100102000 3064K rwx-- [ heap ]
    0000000100400000 4096K rwx-- [ heap ]
    --------------------------

    After 2nd malloc():
    RSS
    9224
    0000000100102000 3064K rwx-- [ heap ]
    0000000100400000 8192K rwx-- [ heap ]
    --------------------------

    After 2nd free():
    RSS
    9224
    0000000100102000 3064K rwx-- [ heap ]
    0000000100400000 8192K rwx-- [ heap ]
    --------------------------

    After 3rd malloc():
    RSS
    13320
    0000000100102000 3064K rwx-- [ heap ]
    0000000100400000 16384K rwx-- [ heap ]

    After 3rd free():
    RSS
    13320
    0000000100102000 3064K rwx-- [ heap ]
    0000000100400000 16384K rwx-- [ heap ]

    before the exit of the program (around 10 minutes after the 3rd free):
    RSS
    13320
    0000000100102000 3064K rwx-- [ heap ]
    0000000100400000 16384K rwx-- [ heap ]

    So, your conclusion would be if you do not have the source: This
    program is a "crap".

    To sum-up the heap in-use reported is rather the touched memory pages
    and released afterwards. However, for some obscure reasons pmap does
    not report the decrease after the free().

    Is there any reliable way to calculate precisely the heap memory in
    use?


  6. Re: Need to know if my application is leaking or not

    abd_web wrote:
    > On 2 nov, 11:59, Spoon wrote:
    >> William Xu wrote:
    >>> Maybe you could try the `valgrind' tool for checking leaks, which is
    >>> very easy to use.

    >> He used Purify, which is functionally similar.http://en.wikipedia.org/wiki/IBM_Rational_Purify

    >
    > If you build and run the following program which is BTW not leaking
    > memory at all and monitor the heap using 'pmap | grep heap':
    >
    > #include
    > #include
    > #include
    >
    > int main ()
    > {
    > char *buffer1;
    > char *buffer2;
    > char *buffer3;
    > int size;
    > char* str =
    > "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxx";
    > size = strlen(str);
    >
    > printf ("\nSleeping 10 sec before 1st allocation\n");
    > sleep(10);
    > buffer1 = (char*) malloc (1024*1024*4*sizeof(char));
    > buffer1 = strncpy(buffer1, str, size);
    > printf ("Sleeping 10 sec after 1st allocation\n");
    > sleep(10);
    >
    > free (buffer1);
    > printf ("Sleeping 10 sec after 1st free\n");
    > sleep(10);
    >
    > buffer2 = (char*) malloc (1024*1024*8*sizeof(char));
    > strncpy(buffer2, str, size);
    >
    > printf ("Sleeping 10 sec after 2nd allocation\n");
    > sleep(10);
    >
    > free (buffer2);
    > printf ("Sleeping 10 sec after 2nd free\n");
    > sleep(10);
    >
    > buffer3 = (char*) malloc (1024*1024*16*sizeof(char));
    > strncpy(buffer3, str, size);
    >
    > printf ("Sleeping 10 sec after 3rd allocation\n");
    > sleep(10);
    >
    > free (buffer3);
    > printf ("Sleeping 1000 sec after 3rd free\n");
    > sleep(1000);
    >
    >
    > printf ("END\n");
    > return 0;
    > }
    >
    > you will see something like:
    >
    > Before any malloc()
    > RSS
    > 1032
    > 0000000100102000 56K rwx-- [ heap ]
    > --------------------------
    >
    > After 1st malloc():
    > RSS
    > 5128
    > 0000000100102000 3064K rwx-- [ heap ]
    > 0000000100400000 4096K rwx-- [ heap ]
    > --------------------------
    >
    > After 1st free():
    > RSS
    > 5128
    > 0000000100102000 3064K rwx-- [ heap ]
    > 0000000100400000 4096K rwx-- [ heap ]
    > --------------------------
    >
    > After 2nd malloc():
    > RSS
    > 9224
    > 0000000100102000 3064K rwx-- [ heap ]
    > 0000000100400000 8192K rwx-- [ heap ]
    > --------------------------
    >
    > After 2nd free():
    > RSS
    > 9224
    > 0000000100102000 3064K rwx-- [ heap ]
    > 0000000100400000 8192K rwx-- [ heap ]
    > --------------------------
    >
    > After 3rd malloc():
    > RSS
    > 13320
    > 0000000100102000 3064K rwx-- [ heap ]
    > 0000000100400000 16384K rwx-- [ heap ]
    >
    > After 3rd free():
    > RSS
    > 13320
    > 0000000100102000 3064K rwx-- [ heap ]
    > 0000000100400000 16384K rwx-- [ heap ]
    >
    > before the exit of the program (around 10 minutes after the 3rd free):
    > RSS
    > 13320
    > 0000000100102000 3064K rwx-- [ heap ]
    > 0000000100400000 16384K rwx-- [ heap ]
    >
    > So, your conclusion would be if you do not have the source: This
    > program is a "crap".
    >
    > To sum-up the heap in-use reported is rather the touched memory pages
    > and released afterwards. However, for some obscure reasons pmap does
    > not report the decrease after the free().
    >
    > Is there any reliable way to calculate precisely the heap memory in
    > use?
    >

    The C RunTime does not release memory back to the OS when 'free' is
    called. Instead it keeps the memory available for future allocation
    attempts. However; since each of your allocation is larger then the free
    pool it has to allocate memory from the OS to grow.

    If you want to release memory back to the OS you would need to utilize
    your own allocation mechanism, for example mmap.

  7. Re: Need to know if my application is leaking or not

    On 2007-11-02, Jim Marshall wrote:
    [...]
    > The C RunTime does not release memory back to the OS when 'free' is
    > called. Instead it keeps the memory available for future allocation
    > attempts. However; since each of your allocation is larger then the free
    > pool it has to allocate memory from the OS to grow.
    >
    > If you want to release memory back to the OS you would need to utilize
    > your own allocation mechanism, for example mmap.


    But why would you want this anyway? It's not real memory. It's only
    virtual address space. So it can stay big. If you want large address
    space for short time only, to avoid dumping it to swap, then you may use
    mmap of files. Again, it all depends on the software you write.


    --
    Minds, like parachutes, function best when open

  8. Re: Need to know if my application is leaking or not

    Andrei Voropaev writes:
    > On 2007-11-02, Jim Marshall wrote:
    > [...]
    >> The C RunTime does not release memory back to the OS when 'free' is
    >> called. Instead it keeps the memory available for future allocation
    >> attempts. However; since each of your allocation is larger then the free
    >> pool it has to allocate memory from the OS to grow.
    >>
    >> If you want to release memory back to the OS you would need to utilize
    >> your own allocation mechanism, for example mmap.

    >
    > But why would you want this anyway? It's not real memory. It's only
    > virtual address space. So it can stay big.


    Depending in the OS which is used (eg Solaris), a 'big virtual address
    space' will correspond with an equally big amount of reserved backing
    store for this address space. Even when no backing store is explicitly
    reserved, the kernel will need to write the content dirty 'reserve
    pages' to a paging area before it can reuse the memory for another
    purpose. Embedded devices large enough to run a full-blown general
    purpose OS often don't have any secondary storage which could be used
    for paging, meaning 'reserve pages' necessarily consume physical
    memory inavailable for other uses.

+ Reply to Thread