all memory used after booting - Setup

This is a discussion on all memory used after booting - Setup ; Hi, Funny thing. I have problems with a newly install RHEL5.1 system using up all memory after boot. System is IBM x-series 3650, with 18 Gigs of RAM installed. After boot, even when booting into single user modus all memory ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: all memory used after booting

  1. all memory used after booting

    Hi,

    Funny thing. I have problems with a newly install RHEL5.1 system using
    up all memory after boot.

    System is IBM x-series 3650, with 18 Gigs of RAM installed.

    After boot, even when booting into single user modus all memory is used
    up:

    [root@tux100p039 ~]# free -m
    total used free shared buffers cached
    Mem: 18047 17944 102 0 3 17
    -/+ buffers/cache: 17924 123
    Swap: 2047 14 2032

    If I boot with "mem=8000", all 8 Gigs are used. If I boot with
    "mem=2000", 2 Gigs are fully used.

    When I boot the RHEL5 rescue system, start network and mount the
    installed system: about 200M are used, so almost all of the 18Gig free.

    When I use the console during installation of the system (reinstalled
    several times), I can see the same kernel is used as in the final system
    (2.6.18-53.el5, x86_64), and 2 Gigs are used, 16 Gigs free. So the
    hardware should be ok I guess.

    We have other 3650 machines with up to 36Gig RAM that do not have this
    problems.

    Of course, there are no obvious processes (ps -eo
    pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=vsize) eating the memory.

    Does anyone have an idea? Does this help?

    [root@tux100p039 ~]# cat /proc/iomem
    00000000-0009abff : System RAM
    00000000-00000000 : Crash kernel
    0009ac00-0009ffff : reserved
    000a0000-000bffff : Video RAM area
    000c0000-000cafff : Video ROM
    000cb000-000cc7ff : Adapter ROM
    000cc800-000cdfff : Adapter ROM
    000d3000-000d31ff : Adapter ROM
    000f0000-000fffff : System ROM
    00100000-bffc74bf : System RAM
    00200000-00460b8c : Kernel code
    00460b8d-00595af7 : Kernel data
    bffc74c0-bffceabf : ACPI Tables
    bffceac0-bfffffff : reserved
    c2000000-c21fffff : PCI Bus #1a
    c2000000-c21fffff : PCI Bus #1b
    c2000000-c20fffff : PCI Bus #1c
    c2100000-c21fffff : PCI Bus #24
    c2100000-c213ffff : 0000:24:00.0
    c4000000-c7efffff : PCI Bus #1a
    c4000000-c7efffff : PCI Bus #1b
    c4000000-c5ffffff : PCI Bus #24
    c5ffc000-c5ffffff : 0000:24:00.0
    c5ffc000-c5ffffff : qla2xxx
    c6000000-c7efffff : PCI Bus #1c
    c7efc000-c7efffff : 0000:1c:00.0
    c7efc000-c7efffff : qla2xxx
    c7f00000-c7ffffff : PCI Bus #04
    c7f00000-c7f07fff : 0000:04:00.0
    c7fe0000-c7ffffff : 0000:04:00.0
    c8000000-c9ffffff : PCI Bus #04
    c9e00000-c9ffffff : 0000:04:00.0
    ca000000-ccffffff : PCI Bus #05
    ca000000-ccffffff : PCI Bus #06
    ca000000-cbffffff : 0000:06:00.0
    ca000000-cbffffff : bnx2
    cd000000-cfffffff : PCI Bus #02
    cd000000-cfffffff : PCI Bus #03
    ce000000-cfffffff : 0000:03:00.0
    ce000000-cfffffff : bnx2
    d0000000-ddffffff : PCI Bus #01
    d0000000-d7ffffff : 0000:01:06.0
    d8000000-d801ffff : 0000:01:06.0
    de000000-dfffffff : PCI Bus #01
    dfff0000-dfffffff : 0000:01:06.0
    e0000000-efffffff : reserved
    f9000000-f90003ff : 0000:00:1d.7
    f9000000-f90003ff : ehci_hcd
    fe700000-fe7003ff : 0000:00:08.0
    fec00000-ffffffff : reserved
    100000000-4bfffffff : System RAM

    Regards,

    Kai.



  2. Re: all memory used after booting

    Kai-Uwe Nielsen wrote:
    > Hi,
    >
    > Funny thing. I have problems with a newly install RHEL5.1 system using
    > up all memory after boot.


    RHEL 5.1 is a little old. They are up to RHEL 5.2 and kernel
    2.6.18-92.1.6.el5PAE. This is almost certainly not the problem, though.

    Immediately after boot, or a while later? I have 8 GB RAM, and when I first
    boot the machine, it takes only about 1/8 of the memory, but a day later it
    takes almost all. But if I look at how it is being used, I get:

    $ free -m
    total used free shared buffers cached
    Mem: 7993 7666 326 0 158 6467
    -/+ buffers/cache: 1041 6952
    Swap: 4000 0 3999

    >
    > System is IBM x-series 3650, with 18 Gigs of RAM installed.
    >
    > After boot, even when booting into single user modus all memory is used
    > up:
    >
    > [root@tux100p039 ~]# free -m
    > total used free shared buffers cached
    > Mem: 18047 17944 102 0 3 17
    > -/+ buffers/cache: 17924 123
    > Swap: 2047 14 2032
    >
    > If I boot with "mem=8000", all 8 Gigs are used. If I boot with
    > "mem=2000", 2 Gigs are fully used.
    >
    > When I boot the RHEL5 rescue system, start network and mount the
    > installed system: about 200M are used, so almost all of the 18Gig free.
    >
    > When I use the console during installation of the system (reinstalled
    > several times), I can see the same kernel is used as in the final system
    > (2.6.18-53.el5, x86_64), and 2 Gigs are used, 16 Gigs free. So the
    > hardware should be ok I guess.


    I wonder if you configured it correctly -- just grabbing at straws here, not
    accusing anyone of bungling.
    >
    > We have other 3650 machines with up to 36Gig RAM that do not have this
    > problems.
    >
    > Of course, there are no obvious processes (ps -eo
    > pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=vsize) eating the memory.


    That command does not work for me.

    I would use top -b -n 1 | less and examine which processes are hogging the
    memory. If you set up top to display memory usage and sort it in order of
    memory usage, you would see if there were either one process hogging it, or
    an unusually large number of processes all using a lesser amount but adding up.

    I get stuff like this:

    top - 08:10:00 up 2 days, 12:58, 6 users, load average: 4.05, 4.09, 4.13
    Tasks: 179 total, 5 running, 174 sleeping, 0 stopped, 0 zombie
    Cpu0 : 2.9%us, 3.8%sy, 90.7%ni, 2.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Cpu1 : 3.0%us, 3.8%sy, 90.5%ni, 2.6%id, 0.0%wa, 0.1%hi, 0.1%si, 0.0%st
    Cpu2 : 2.4%us, 3.9%sy, 91.7%ni, 2.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Cpu3 : 2.3%us, 3.7%sy, 91.9%ni, 2.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 8185240k total, 7795916k used, 389324k free, 164440k buffers
    Swap: 4096496k total, 580k used, 4095916k free, 6649424k cached

    PID PPID USER PR NI S VIRT RES SHR SWAP %MEM %CPU TIME+ P
    3663 3659 postgres 15 0 S 2010m 766m 766m 1.2g 9.6 0 21:19.88 1
    3897 3681 boinc 39 19 R 947m 306m 7912 641m 3.8 92 3460:45 3
    24682 24677 jdbeyer 18 0 S 207m 101m 20m 105m 1.3 0 3:06.85 0
    3879 3680 boinc 39 19 R 202m 91m 4120 110m 1.1 98 3460:11 1
    24951 24937 jdbeyer 15 0 S 222m 89m 23m 133m 1.1 0 7:51.02 0

    >
    > Does anyone have an idea? Does this help?
    >

    [snip]
    --
    .~. Jean-David Beyer Registered Linux User 85642.
    /V\ PGP-Key: 9A2FC99A Registered Machine 241939.
    /( )\ Shrewsbury, New Jersey http://counter.li.org
    ^^-^^ 07:50:01 up 2 days, 12:38, 5 users, load average: 4.11, 4.19, 4.18

  3. Re: all memory used after booting

    Hi,

    * Jean-David Beyer :
    >
    > Immediately after boot, or a while later? I have 8 GB RAM, and when I first
    > boot the machine, it takes only about 1/8 of the memory, but a day later...


    No, _immediately_ after boot. Even when booting to single user modus.

    > I wonder if you configured it correctly -- just grabbing at straws here, not
    > accusing anyone of bungling.


    Well, let me put it this way: We went from "our own tuned system
    configuration" down to "pretty much fire and install" with no special
    configuration.

    I just looked at another x3650 with 16 Gigs that was installed from
    pretty much the same rhel5 kickstart template: since nothing much is running on
    that machine, it still reports 13 Gigs of _free_ mem after days. We tend
    to oversize these servers. ;-)

    I know the difference between free mem and cache mem used for
    filesystem. Well, since we're talking about a bare installation, the
    problematic machine could suck in it's own filesystem twice into memory
    and still should have more than 10 Gigs of RAM free. :-)

    sosreport to Redhat is on it's way...

    Regards,

    Kai.


  4. Re: all memory used after booting

    Kai-Uwe Nielsen wrote:
    > Hi,
    >
    > * Jean-David Beyer :
    >> Immediately after boot, or a while later? I have 8 GB RAM, and when I first
    >> boot the machine, it takes only about 1/8 of the memory, but a day later...

    >
    > No, _immediately_ after boot. Even when booting to single user modus.
    >
    >> I wonder if you configured it correctly -- just grabbing at straws here, not
    >> accusing anyone of bungling.

    >
    > Well, let me put it this way: We went from "our own tuned system
    > configuration" down to "pretty much fire and install" with no special
    > configuration.
    >
    > I just looked at another x3650 with 16 Gigs that was installed from
    > pretty much the same rhel5 kickstart template: since nothing much is running on
    > that machine, it still reports 13 Gigs of _free_ mem after days. We tend
    > to oversize these servers. ;-)
    >
    > I know the difference between free mem and cache mem used for
    > filesystem. Well, since we're talking about a bare installation, the
    > problematic machine could suck in it's own filesystem twice into memory
    > and still should have more than 10 Gigs of RAM free. :-)
    >
    > sosreport to Redhat is on it's way...
    >

    Good idea. You are paying for support; you might as well get some.


    --
    .~. Jean-David Beyer Registered Linux User 85642.
    /V\ PGP-Key: 9A2FC99A Registered Machine 241939.
    /( )\ Shrewsbury, New Jersey http://counter.li.org
    ^^-^^ 09:00:01 up 2 days, 13:48, 5 users, load average: 4.06, 4.06, 4.07

  5. Re: all memory used after booting

    Kai-Uwe Nielsen wrote:
    > Hi,
    >
    > * Jean-David Beyer :
    >> Immediately after boot, or a while later? I have 8 GB RAM, and when I first
    >> boot the machine, it takes only about 1/8 of the memory, but a day later...

    >
    > No, _immediately_ after boot. Even when booting to single user modus.
    >

    Perhaps it's just grabbed it all for disk cache..



  6. Re: all memory used after booting

    * The Natural Philosopher :
    > Kai-Uwe Nielsen wrote:
    > > Hi,
    > >
    > > * Jean-David Beyer :
    > >> Immediately after boot, or a while later? I have 8 GB RAM, and when I first
    > >> boot the machine, it takes only about 1/8 of the memory, but a day later...

    > >
    > > No, _immediately_ after boot. Even when booting to single user modus.
    > >

    > Perhaps it's just grabbed it all for disk cache..


    Yeah, 18 Gig grabbed for a filesystem thats less than 2 Gig
    altogether...

    But this thread can be closed. I have been stupid and didn't see the
    obvious problem. It just came to my mind _after_ starting this thread.
    After several days of struggling.

    Well.

    We use kickstart files for installation.
    We do have certain templates.
    We had an Oracle Consultant inhouse to optimize our database machines.
    He did that and told us some additional stuff to be included in our
    Oracle template.
    This machine shall be a small standalone Oracle machine.
    The settings were optimized for 64Gig machines.

    So the story goes... And all the time I forgot that the Oracle include
    in the template not only creates users and stuff, but also manipulates
    /etc/sysctl.conf

    I really should have found out sooner, stupid me. Just for sake of
    completeness, this was the setting that grabbed all the memory:

    [root@tux100p039 /etc]# free -m
    total used free shared buffers cached
    Mem: 18047 232 17815 0 18 150
    -/+ buffers/cache: 62 17985
    Swap: 2047 0 2047
    [root@tux100p039 /etc]# sysctl vm.nr_hugepages=10000
    vm.nr_hugepages = 10000
    [root@tux100p039 /etc]# free -m
    total used free shared buffers cached
    Mem: 18047 17950 97 0 18 143
    -/+ buffers/cache: 17788 258
    Swap: 2047 0 2047

    Now I go hide somewhere... ;-)


    Regards,

    Kai.



  7. Re: all memory used after booting

    Kai-Uwe Nielsen wrote:

    > But this thread can be closed. I have been stupid and didn't see the
    > obvious problem. It just came to my mind _after_ starting this thread.
    > After several days of struggling.
    >
    > Well.
    >
    > We use kickstart files for installation.
    > We do have certain templates.
    > We had an Oracle Consultant inhouse to optimize our database machines.
    > He did that and told us some additional stuff to be included in our
    > Oracle template.
    > This machine shall be a small standalone Oracle machine.
    > The settings were optimized for 64Gig machines.
    >
    > So the story goes... And all the time I forgot that the Oracle include
    > in the template not only creates users and stuff, but also manipulates
    > /etc/sysctl.conf
    >
    > I really should have found out sooner, stupid me. Just for sake of
    > completeness, this was the setting that grabbed all the memory:
    >
    > [root@tux100p039 /etc]# free -m
    > total used free shared buffers cached
    > Mem: 18047 232 17815 0 18 150
    > -/+ buffers/cache: 62 17985
    > Swap: 2047 0 2047
    > [root@tux100p039 /etc]# sysctl vm.nr_hugepages=10000
    > vm.nr_hugepages = 10000
    > [root@tux100p039 /etc]# free -m
    > total used free shared buffers cached
    > Mem: 18047 17950 97 0 18 143
    > -/+ buffers/cache: 17788 258
    > Swap: 2047 0 2047
    >
    > Now I go hide somewhere... ;-)
    >

    Right! You are alone. No one else here has ever overlooked something that
    is only obvious in retrospect. ;-)

    --
    .~. Jean-David Beyer Registered Linux User 85642.
    /V\ PGP-Key: 9A2FC99A Registered Machine 241939.
    /( )\ Shrewsbury, New Jersey http://counter.li.org
    ^^-^^ 10:30:01 up 2 days, 15:18, 5 users, load average: 4.35, 4.31, 4.27

  8. Re: all memory used after booting

    Kai-Uwe Nielsen wrote:
    > * The Natural Philosopher :
    >> Kai-Uwe Nielsen wrote:
    >>> Hi,
    >>>
    >>> * Jean-David Beyer :
    >>>> Immediately after boot, or a while later? I have 8 GB RAM, and when I first
    >>>> boot the machine, it takes only about 1/8 of the memory, but a day later...
    >>> No, _immediately_ after boot. Even when booting to single user modus.
    >>>

    >> Perhaps it's just grabbed it all for disk cache..

    >
    > Yeah, 18 Gig grabbed for a filesystem thats less than 2 Gig
    > altogether...
    >
    > But this thread can be closed. I have been stupid and didn't see the
    > obvious problem. It just came to my mind _after_ starting this thread.
    > After several days of struggling.
    >
    > Well.
    >
    > We use kickstart files for installation.
    > We do have certain templates.
    > We had an Oracle Consultant inhouse to optimize our database machines.
    > He did that and told us some additional stuff to be included in our
    > Oracle template.
    > This machine shall be a small standalone Oracle machine.
    > The settings were optimized for 64Gig machines.
    >
    > So the story goes... And all the time I forgot that the Oracle include
    > in the template not only creates users and stuff, but also manipulates
    > /etc/sysctl.conf
    >
    > I really should have found out sooner, stupid me. Just for sake of
    > completeness, this was the setting that grabbed all the memory:
    >
    > [root@tux100p039 /etc]# free -m
    > total used free shared buffers cached
    > Mem: 18047 232 17815 0 18 150
    > -/+ buffers/cache: 62 17985
    > Swap: 2047 0 2047
    > [root@tux100p039 /etc]# sysctl vm.nr_hugepages=10000
    > vm.nr_hugepages = 10000
    > [root@tux100p039 /etc]# free -m
    > total used free shared buffers cached
    > Mem: 18047 17950 97 0 18 143
    > -/+ buffers/cache: 17788 258
    > Swap: 2047 0 2047
    >
    > Now I go hide somewhere... ;-)
    >


    Yeah..i've done a days consulatncy to discover the bug in the template
    that broke everyone's network connection..after we had spent three
    defining exactly what it was to be, and the manager had our original,
    and had to admit that was NOT what was being installed.

    ;-)

    Any consultant/sysadmin can recite you similar stories too..

    Like the one who used EMACS to edit a config file with three hard links
    to it. Emacs renamed it to something.old,(or something) and created a
    new one of the original name, to which nothing pointed any more...

    >
    > Regards,
    >
    > Kai.
    >
    >


  9. Re: all memory used after booting

    Kai-Uwe Nielsen writes:

    >Hi,


    >Funny thing. I have problems with a newly install RHEL5.1 system using
    >up all memory after boot.


    Not sure what you mean "used up". Linux uses spare memory as a cache for
    old programs/data, just in case you want them again. Speeds up loading. Why
    in the world do you want spare memory just sitting there when it could do
    something useful. If some running program needs the memory that is being
    used for caching, that cached stuff is thrown away and the menory handed
    over to what needs it, so there is no penalty.


    >System is IBM x-series 3650, with 18 Gigs of RAM installed.


    >After boot, even when booting into single user modus all memory is used
    >up:


    >[root@tux100p039 ~]# free -m
    > total used free shared buffers cached
    >Mem: 18047 17944 102 0 3 17
    >-/+ buffers/cache: 17924 123
    >Swap: 2047 14 2032



    >If I boot with "mem=8000", all 8 Gigs are used. If I boot with
    >"mem=2000", 2 Gigs are fully used.


    >When I boot the RHEL5 rescue system, start network and mount the
    >installed system: about 200M are used, so almost all of the 18Gig free.


    >When I use the console during installation of the system (reinstalled
    >several times), I can see the same kernel is used as in the final system
    >(2.6.18-53.el5, x86_64), and 2 Gigs are used, 16 Gigs free. So the
    >hardware should be ok I guess.


    >We have other 3650 machines with up to 36Gig RAM that do not have this
    >problems.


    I assume these ar x86_64 systems.


    >Of course, there are no obvious processes (ps -eo
    >pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=vsize) eating the memory.


    >Does anyone have an idea? Does this help?


    >[root@tux100p039 ~]# cat /proc/iomem
    >00000000-0009abff : System RAM
    > 00000000-00000000 : Crash kernel
    >0009ac00-0009ffff : reserved
    >000a0000-000bffff : Video RAM area
    >000c0000-000cafff : Video ROM
    >000cb000-000cc7ff : Adapter ROM
    >000cc800-000cdfff : Adapter ROM
    >000d3000-000d31ff : Adapter ROM
    >000f0000-000fffff : System ROM
    >00100000-bffc74bf : System RAM
    > 00200000-00460b8c : Kernel code
    > 00460b8d-00595af7 : Kernel data
    >bffc74c0-bffceabf : ACPI Tables
    >bffceac0-bfffffff : reserved
    >c2000000-c21fffff : PCI Bus #1a
    > c2000000-c21fffff : PCI Bus #1b
    > c2000000-c20fffff : PCI Bus #1c
    > c2100000-c21fffff : PCI Bus #24
    > c2100000-c213ffff : 0000:24:00.0
    >c4000000-c7efffff : PCI Bus #1a
    > c4000000-c7efffff : PCI Bus #1b
    > c4000000-c5ffffff : PCI Bus #24
    > c5ffc000-c5ffffff : 0000:24:00.0
    > c5ffc000-c5ffffff : qla2xxx
    > c6000000-c7efffff : PCI Bus #1c
    > c7efc000-c7efffff : 0000:1c:00.0
    > c7efc000-c7efffff : qla2xxx
    >c7f00000-c7ffffff : PCI Bus #04
    > c7f00000-c7f07fff : 0000:04:00.0
    > c7fe0000-c7ffffff : 0000:04:00.0
    >c8000000-c9ffffff : PCI Bus #04
    > c9e00000-c9ffffff : 0000:04:00.0
    >ca000000-ccffffff : PCI Bus #05
    > ca000000-ccffffff : PCI Bus #06
    > ca000000-cbffffff : 0000:06:00.0
    > ca000000-cbffffff : bnx2
    >cd000000-cfffffff : PCI Bus #02
    > cd000000-cfffffff : PCI Bus #03
    > ce000000-cfffffff : 0000:03:00.0
    > ce000000-cfffffff : bnx2
    >d0000000-ddffffff : PCI Bus #01
    > d0000000-d7ffffff : 0000:01:06.0
    > d8000000-d801ffff : 0000:01:06.0
    >de000000-dfffffff : PCI Bus #01
    > dfff0000-dfffffff : 0000:01:06.0
    >e0000000-efffffff : reserved
    >f9000000-f90003ff : 0000:00:1d.7
    > f9000000-f90003ff : ehci_hcd
    >fe700000-fe7003ff : 0000:00:08.0
    >fec00000-ffffffff : reserved
    >100000000-4bfffffff : System RAM


    >Regards,


    > Kai.




+ Reply to Thread