using FILE OPEN quadruples memory usage! - Protocols

This is a discussion on using FILE OPEN quadruples memory usage! - Protocols ; I have discovered that using FILE OPEN in C-Kermit 8.0.211 causes Kermit's memory usage to quadruple. The memory usage does not go back down after the file(s) are closed, either. This memory explosion does NOT appear to occur when the ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: using FILE OPEN quadruples memory usage!

  1. using FILE OPEN quadruples memory usage!

    I have discovered that using FILE OPEN in C-Kermit 8.0.211 causes
    Kermit's memory usage to quadruple. The memory usage does not go back
    down after the file(s) are closed, either.

    This memory explosion does NOT appear to occur when the "old" OPEN FILE
    method is used, only with the "new" FILE OPEN method. So it is
    possible to work around it, albeit inconveniently at times.

    Further openings of files do not appear to result in further memory
    expansion. I was able to have multiple files open using FILE OPEN and
    it did not proceed to octuple memory usage, for example.

    I'm using Kermit compiled for ARM with "make linuxnc KFLAGS=-DNOBIGBUF"
    (I'm trying to save memory!) and running on a 32M embedded board
    running the Linux 2.4.26-ts9 kernel.

    How can this be happening if FILE OPEN is simply a front-end for the
    fopen() system call? I could see memory on the order of kilobytes
    being allocated, but we're talking several megabytes of memory being
    consumed here, just from opening a text file.

    Is there perhaps some Makefile option (along the lines of the
    -DNOBIGBUF that I already used) that can mitigate this?

    Witness the following test: session one is a Kermit session, and
    session 2 is showing the results of ps v -p performed on the Kermit
    process at each step.

    --- session one ---
    $ kermit
    C-Kermit 8.0.211, 10 Apr 2004, for Linux
    Copyright (C) 1985, 2004,
    Trustees of Columbia University in the City of New York.
    Type ? or HELP for help.
    (/home/dlogger/) C-Kermit>echo \v(pid)
    29009
    (/home/dlogger/) C-Kermit>

    -- session two --
    $ ps v -p 29009
    PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
    29009 pts/1 S 0:00 247 1666 2849 1308 4.5 kermit
    $

    -- session one --
    (/home/dlogger/) C-Kermit>file open \%f .profile
    (/home/dlogger/) C-Kermit>

    -- session two --
    $ ps v -p 29009
    PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
    29009 pts/1 S 0:00 265 1666 6905 5452 18.9 kermit
    $

    -- session one --
    (/home/dlogger/) C-Kermit>file close all
    (/home/dlogger/) C-Kermit>

    -- session two --
    $ ps v -p 29009
    PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
    29009 pts/1 S 0:00 267 1666 6905 5460 19.0 kermit
    $
    --- end of sessions ---

    Note that the memory usage actually went UP even further after the file
    was closed!


  2. Re: using FILE OPEN quadruples memory usage!

    On 2006-05-04, tomviolin wrote:
    : I have discovered that using FILE OPEN in C-Kermit 8.0.211 causes
    : Kermit's memory usage to quadruple. The memory usage does not go back
    : down after the file(s) are closed, either.
    :
    : This memory explosion does NOT appear to occur when the "old" OPEN FILE
    : method is used, only with the "new" FILE OPEN method. So it is
    : possible to work around it, albeit inconveniently at times.
    :
    : Further openings of files do not appear to result in further memory
    : expansion. I was able to have multiple files open using FILE OPEN and
    : it did not proceed to octuple memory usage, for example.
    :
    The FILE OPEN code is in ckuus7.c within #ifdef CKCHANNELIO..#endif,
    about 1640 lines that I wrote six years ago. There's a spot where we
    malloc z_maxchan times sizeof(struct ckz_file), which includes a filename
    buffer of about 1K. This happens once, the first time FILE OPEN is used.
    Probably this storage is never deallocated, since a new file might be
    opened at any time. When I wrote the code, the maximum number of open files
    per process (as reported by sysconf) was usually a small number, less than
    100. Now I see that in current Linuxes, it's more like 1000. So that would
    account for about a megabyte.

    : I'm using Kermit compiled for ARM with "make linuxnc KFLAGS=-DNOBIGBUF"
    : (I'm trying to save memory!) and running on a 32M embedded board
    : running the Linux 2.4.26-ts9 kernel.
    :
    : How can this be happening if FILE OPEN is simply a front-end for the
    : fopen() system call?
    :
    It does its own bookkeeping, has its own mini-FILE struct for each channel.

    : I could see memory on the order of kilobytes
    : being allocated, but we're talking several megabytes of memory being
    : consumed here, just from opening a text file.
    :
    In Linux (in this case Red Hat AS 4 on AMD-64), if I start Kermit, log debug,
    do an FOPEN, exit, and then "grep z_maxchan debug.log" I see:

    z_open z_maxchan 1=-2
    z_open z_maxchan 2=1006
    z_open z_maxchan 3=1006

    1006 is the number of mini-FILE structs that are allocated. I can see that
    this could be done with a big more finesse. It wasn't an issue before. I'll
    see what I can do in the next 8.0.212 development upload:

    http://www.columbia.edu/kermit/ckdaily.html

    - Frank

  3. Re: using FILE OPEN quadruples memory usage!

    On 2006-05-05, Frank da Cruz wrote:
    : On 2006-05-04, tomviolin wrote:
    :: I have discovered that using FILE OPEN in C-Kermit 8.0.211 causes
    :: Kermit's memory usage to quadruple. The memory usage does not go back
    :: down after the file(s) are closed, either.
    : ...
    : I'll see what I can do in the next 8.0.212 development upload:
    :
    : http://www.columbia.edu/kermit/ckdaily.html
    :
    If you have the most recent "ckdaily", please download a new copy of
    ckuus7.c from here:

    ftp://kermit.columbia.edu/kermit/tmp/ckuus7.c

    and rebuild, see if it's better. I ran a few scripts through it that open
    and close and read files and they seem to be ok.

    - Frank

  4. Re: using FILE OPEN quadruples memory usage!

    Frank,

    Thanks for doing this so quickly! Yes, the memory usage is much
    improved now.

    -Tom


+ Reply to Thread