vm.kmem_size settings doesn't affect loader? - FreeBSD

This is a discussion on vm.kmem_size settings doesn't affect loader? - FreeBSD ; Today I've experienced zfs-related kernel panic. Log says: savecore: reboot after panic: kmem_malloc(131072): kmem_map too small: 327684096 total allocated Reported amount of memory (327684096) is wrong, because i made suggested tuning in my loader.conf: vm.kmem_size="512M" vm.kmem_size_max="512M" Just to be sure: ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: vm.kmem_size settings doesn't affect loader?

  1. vm.kmem_size settings doesn't affect loader?



    Today I've experienced zfs-related kernel panic. Log says:

    savecore: reboot after panic: kmem_malloc(131072): kmem_map too
    small: 327684096 total allocated

    Reported amount of memory (327684096) is wrong, because i made suggested
    tuning in my loader.conf:

    vm.kmem_size="512M"
    vm.kmem_size_max="512M"

    Just to be sure:

    # sysctl vm | grep kmem
    vm.kmem_size: 536870912
    vm.kmem_size_min: 0
    vm.kmem_size_max: 536870912
    vm.kmem_size_scale: 3

    Am I missing something?

    --
    Bartosz Stec

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  2. Re: vm.kmem_size settings doesn't affect loader?

    On Thursday 25 September 2008 06:26:58 am Bartosz Stec wrote:
    >
    > Today I've experienced zfs-related kernel panic. Log says:
    >
    > savecore: reboot after panic: kmem_malloc(131072): kmem_map too
    > small: 327684096 total allocated
    >
    > Reported amount of memory (327684096) is wrong, because i made suggested
    > tuning in my loader.conf:
    >
    > vm.kmem_size="512M"
    > vm.kmem_size_max="512M"
    >
    > Just to be sure:
    >
    > # sysctl vm | grep kmem
    > vm.kmem_size: 536870912
    > vm.kmem_size_min: 0
    > vm.kmem_size_max: 536870912
    > vm.kmem_size_scale: 3
    >
    > Am I missing something?


    If kvm is fragmented you could have a malloc fail even if there are enough
    total free bytes for the allocatoin.

    --
    John Baldwin
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  3. Re: vm.kmem_size settings doesn't affect loader?

    On Thu, Sep 25, 2008 at 12:26:58PM +0200, Bartosz Stec wrote:
    >
    >
    > Today I've experienced zfs-related kernel panic. Log says:
    >
    > savecore: reboot after panic: kmem_malloc(131072): kmem_map too
    > small: 327684096 total allocated
    >
    > Reported amount of memory (327684096) is wrong, because i made suggested
    > tuning in my loader.conf:
    >
    > vm.kmem_size="512M"
    > vm.kmem_size_max="512M"
    >
    > Just to be sure:
    >
    > # sysctl vm | grep kmem
    > vm.kmem_size: 536870912
    > vm.kmem_size_min: 0
    > vm.kmem_size_max: 536870912
    > vm.kmem_size_scale: 3
    >
    > Am I missing something?


    I believe this is normal. The amount shown in "total allocated" will
    not match what you have vm.kmem_size or vm.kmem_size_max set to.
    Someone more familiar with the VM can explain why this is, but as I
    understand it, it's 100% normal.

    Your options are:

    1) Consider increasing it from 512M to something like 1.5GB; do not
    increase it past that on RELENG_7, as there isn't support for more than
    2GB total. For example, on a 1GB memory machine, I often recommend
    768M. On 2GB machines, 1536M. You will need to run -CURRENT if you
    want more.

    2) Tune ZFS aggressively. Start by setting vfs.zfs.arc_min="16M"
    and vfs.zfs.arc_max="64M".

    If your machine has some small amount of memory (768MB, 1GB, etc.),
    then you probably shouldn't be using ZFS.

    --
    | Jeremy Chadwick jdc at parodius.com |
    | Parodius Networking http://www.parodius.com/ |
    | UNIX Systems Administrator Mountain View, CA, USA |
    | Making life hard for others since 1977. PGP: 4BD6C0CB |

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  4. Re: vm.kmem_size settings doesn't affect loader?

    Jeremy Chadwick pisze:
    > On Thu, Sep 25, 2008 at 12:26:58PM +0200, Bartosz Stec wrote:
    >
    >> Today I've experienced zfs-related kernel panic. Log says:
    >>
    >> savecore: reboot after panic: kmem_malloc(131072): kmem_map too
    >> small: 327684096 total allocated
    >>
    >> Reported amount of memory (327684096) is wrong, because i made suggested
    >> tuning in my loader.conf:
    >>
    >> vm.kmem_size="512M"
    >> vm.kmem_size_max="512M"
    >>
    >> Just to be sure:
    >>
    >> # sysctl vm | grep kmem
    >> vm.kmem_size: 536870912
    >> vm.kmem_size_min: 0
    >> vm.kmem_size_max: 536870912
    >> vm.kmem_size_scale: 3
    >>
    >> Am I missing something?
    >>

    >
    > I believe this is normal. The amount shown in "total allocated" will
    > not match what you have vm.kmem_size or vm.kmem_size_max set to.
    > Someone more familiar with the VM can explain why this is, but as I
    > understand it, it's 100% normal.
    >

    Thanks for explaining. This part of tuning guide confused me:

    "I was able to generate the following kernel panic in less than a
    minute by copying files from a linux server connected via gigabit
    crossover:
    Apr 8 06:46:08 nas savecore: reboot after panic: kmem_malloc(131072): kmem_map too small: 528273408 total allocated
    As you can see, this is *with* vm.kmem_size="512M"!"

    I've just expected similiar numbers in my case.
    > Your options are:
    >
    > 1) Consider increasing it from 512M to something like 1.5GB; do not
    > increase it past that on RELENG_7, as there isn't support for more than
    > 2GB total. For example, on a 1GB memory machine, I often recommend
    > 768M. On 2GB machines, 1536M. You will need to run -CURRENT if you
    > want more.
    >
    > 2) Tune ZFS aggressively. Start by setting vfs.zfs.arc_min="16M"
    > and vfs.zfs.arc_max="64M".
    >
    > If your machine has some small amount of memory (768MB, 1GB, etc.),
    > then you probably shouldn't be using ZFS.
    >
    >

    Problem occured on i386 machine with 1GB of memory and 7.1-pre (3HDD,
    40GB, RAIDZ1). I know that i386 is highly unrecommended for ZFS, but
    it's just a home box for testing and learning purposes - I just want to
    know what I'm doing and what should I expect when I decide to put ZFS on
    server machines Currently, from posts on freebsd-fs, I conclude that
    even with a gigs of kmem and using AMD64, we still can experience panic
    from kmem_malloc. Manual tuning is hard for me because I'm not familiar
    with BSD kernel code nor kernel memory management. I'm just an end-user
    who love concepts of ZFS and wait for it to be (more) stable. Of course
    I've followed tuning guide carefully.
    So, for now, I will add another 512MB of memory and increse vm.kmem_size
    and do more tests. Thanks once again for explaining and suggestions.
    Good luck with ZFS everyone!

    --
    Bartosz Stec

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  5. Re: vm.kmem_size settings doesn't affect loader?

    On Thu, Sep 25, 2008 at 04:14:02PM +0200, Bartosz Stec wrote:
    >> Your options are:
    >>
    >> 1) Consider increasing it from 512M to something like 1.5GB; do not
    >> increase it past that on RELENG_7, as there isn't support for more than
    >> 2GB total. For example, on a 1GB memory machine, I often recommend
    >> 768M. On 2GB machines, 1536M. You will need to run -CURRENT if you
    >> want more.
    >>
    >> 2) Tune ZFS aggressively. Start by setting vfs.zfs.arc_min="16M"
    >> and vfs.zfs.arc_max="64M".
    >>
    >> If your machine has some small amount of memory (768MB, 1GB, etc.),
    >> then you probably shouldn't be using ZFS.
    >>
    >>

    > Problem occured on i386 machine with 1GB of memory and 7.1-pre (3HDD,
    > 40GB, RAIDZ1). I know that i386 is highly unrecommended for ZFS, but
    > it's just a home box for testing and learning purposes - I just want to
    > know what I'm doing and what should I expect when I decide to put ZFS on
    > server machines Currently, from posts on freebsd-fs, I conclude that
    > even with a gigs of kmem and using AMD64, we still can experience panic
    > from kmem_malloc.


    The i386 vs. amd64 argument is bogus, if you ask me. ZFS works on both.
    amd64 is recommended because ZFS contains code that makes heavy use of
    64-bit values, and because amd64 offers large amounts of addressed
    memory without disgusting hacks like PAE.

    That said -- yes, even with "gigs of kmem and using AMD64", you can
    still panic due to kmem exhaustion. I have fairly decent experience
    with this problem, because it haunted me for quite some time.

    A large portion of the problem is that kmem_max, on i386 and amd64 (yes,
    you read that right) has a 2GB limit on RELENG_7. I repeat: a 2GB
    limit, regardless of i386 or amd64.

    This limit has been increased to 512GB on CURRENT, but there are no
    plans to MFC those changes, as they are too major.

    Let me tell you something I did this weekend. I had to copy literally
    200GB of data from a ZFS raidz1 pool (spread across 3 disks) to two
    different places: 1) a UFS2 filesystem on a different disk, and 2)
    across a gigE network to a Windows machine. I had to do this because I
    was adding a disk to the vdev, which cannot be done without re-creating
    the pool (this is a known problem with ZFS, and has nothing to do with
    FreeBSD).

    The machine hosting the data runs RELENG_7 with amd64, and contains 4GB
    of memory. However, I've accomplished the same task with only 2GB of
    memory as well.

    These are the tuning settings I use:

    vm.kmem_size="1536M"
    vm.kmem_size_max="1536M"
    vfs.zfs.arc_min="16M"
    vfs.zfs.arc_max="64M"

    The entire copying process took almost 2 hours. Not once did I
    experience kmem exhaustion. I can *guarantee* that I would have crashed
    the box numerous times had I not tuned the machine with the values
    above.

    > Manual tuning is hard for me because I'm not familiar
    > with BSD kernel code nor kernel memory management. I'm just an end-user
    > who love concepts of ZFS and wait for it to be (more) stable. Of course
    > I've followed tuning guide carefully.


    I'm an "experienced" end-user who has very little experience with BSD
    kernel code and absolutely no experience with kernel memory management.
    Proper tuning is all that's needed, regardless of your knowledge set.

    Please try installing 2GB of memory in your i386 box, and then use
    the exact loader.conf values I specified above.

    --
    | Jeremy Chadwick jdc at parodius.com |
    | Parodius Networking http://www.parodius.com/ |
    | UNIX Systems Administrator Mountain View, CA, USA |
    | Making life hard for others since 1977. PGP: 4BD6C0CB |

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  6. Re: vm.kmem_size settings doesn't affect loader?

    > These are the tuning settings I use:
    >
    > vm.kmem_size="1536M"
    > vm.kmem_size_max="1536M"
    > vfs.zfs.arc_min="16M"
    > vfs.zfs.arc_max="64M"


    >
    > The entire copying process took almost 2 hours. Not once did I
    > experience kmem exhaustion. I can *guarantee* that I would have crashed
    > the box numerous times had I not tuned the machine with the values
    > above.


    I am interested in the last two values you have there - I also use ZFS and
    tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered
    toi set the arc_min - and the default appears to be 16M anyway. The arc_max
    value I do set, but I got dreadful performance when I had it at 6M, so
    I upped it to 128M. This has not resulted in any panics, and gives much better
    performance. As I understand it there is only one ARC, and it comes from
    kernel space too, so setting it to an extra 64 meg should not use that much
    more memory, yes ?

    My setup is:

    vm.kmem_size="1024M"
    vm.kmem_size_max="1024M"
    vfs.zfs.arc_max="128M"
    vfs.zfs.prefetch_disable=1

    This is on a 2gb amd64 machine, and works beautifully, with no out of
    memory errors and panics. I have been trying to panic this for a few
    days. I will probably chnage the kmem size to 1536 which is what I use on
    other machines though.

    -pete.
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  7. Re: vm.kmem_size settings doesn't affect loader?

    On Thu, Sep 25, 2008 at 04:10:07PM +0100, Pete French wrote:
    > > These are the tuning settings I use:
    > >
    > > vm.kmem_size="1536M"
    > > vm.kmem_size_max="1536M"
    > > vfs.zfs.arc_min="16M"
    > > vfs.zfs.arc_max="64M"
    > >
    > > The entire copying process took almost 2 hours. Not once did I
    > > experience kmem exhaustion. I can *guarantee* that I would have crashed
    > > the box numerous times had I not tuned the machine with the values
    > > above.

    >
    > I am interested in the last two values you have there - I also use ZFS and
    > tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered
    > toi set the arc_min - and the default appears to be 16M anyway. The arc_max
    > value I do set, but I got dreadful performance when I had it at 6M, so
    > I upped it to 128M.


    ARC tuning is what's necessary to induce ZFS memory constraints. If you
    only set kmem_size/max, you will likely run into cases where the ARC
    grows to such a size that kmem exhaustion happens. Even the Solaris ZFS
    "do not tune ZFS" article outlines this fact (read it all, including the
    RFEs):

    http://www.solarisinternals.com/wiki..._the_ARC_Cache

    Just keep in mind this is specific to Solaris, but the problems
    conceptually are identical to FreeBSD.

    > This has not resulted in any panics, and gives much better
    > performance. As I understand it there is only one ARC, and it comes from
    > kernel space too, so setting it to an extra 64 meg should not use that much
    > more memory, yes ?


    The ARC uses kmem. "Should not use that much more memory" is a matter
    of opinion; if an additional 64MB given to ARC causes kmem exhaustion
    (there's no global standard for this; keep reading, I explain it near
    the bottom of my mail), then decreasing the ARC max is necesary.

    On the bright side, it's very easy to determine how much memory is being
    dedicated to the ARC. sysctl kstat.zfs.misc will disclose this data
    and I believe the following statistics are the ones you have to examine
    for this issue:

    kstat.zfs.misc.arcstats.p
    kstat.zfs.misc.arcstats.c
    kstat.zfs.misc.arcstats.c_min
    kstat.zfs.misc.arcstats.c_max
    kstat.zfs.misc.arcstats.size

    On my system (the one using the tuning parameters you're questioning):

    kstat.zfs.misc.arcstats.p: 60487680
    kstat.zfs.misc.arcstats.c: 67108864
    kstat.zfs.misc.arcstats.c_min: 16777216
    kstat.zfs.misc.arcstats.c_max: 67108864
    kstat.zfs.misc.arcstats.size: 67092480

    For those wanting to read about the ARC and what it's for:

    http://en.wikipedia.org/wiki/Adaptive_Replacement_Cache

    > My setup is:
    >
    > vm.kmem_size="1024M"
    > vm.kmem_size_max="1024M"
    > vfs.zfs.arc_max="128M"
    > vfs.zfs.prefetch_disable=1
    >
    > This is on a 2gb amd64 machine, and works beautifully, with no out of
    > memory errors and panics. I have been trying to panic this for a few
    > days. I will probably chnage the kmem size to 1536 which is what I use on
    > other machines though.


    Disabling prefetch shouldn't have any effect on stability -- many of us
    (including FreeBSD core members) have found that disabling prefetch
    increases system responsiveness during heavy I/O, and results in a less
    "bursty" system. I disable prefetch, but it should have no bearing on
    the issue we're discussing.

    I'm sure the memory tuning values work fine for you, but they do not for
    me; I can absolutely panic the machine with a kmem size of 1GB. How?
    Because it's entirely dependent upon the type of ZFS pools/setup a
    person has, how much actual load they put on ZFS, and probably a hundred
    other things.

    Let's adhere to the K.I.S.S. concept, and not turn this thread into
    a "tuning micro-management" thread.

    What end-users want is ZFS that "just works", which on Solaris 10 it
    does. But I have no qualms with tuning ZFS on FreeBSD, though others
    seem to. A "tuning guide" cannot easily be written because there are
    too many factors to take into consideration. Besides, such a guide
    (documenting "how it all interconnects, memory-wise") would end up being
    tailored to the technical, which won't help end-users. The existing
    "ZFS tuning guide" for FreeBSD keeps things vague for a good reason --
    I'm absolutely 100% positive an extensive document would just confuse
    the living daylights out of people.

    --
    | Jeremy Chadwick jdc at parodius.com |
    | Parodius Networking http://www.parodius.com/ |
    | UNIX Systems Administrator Mountain View, CA, USA |
    | Making life hard for others since 1977. PGP: 4BD6C0CB |

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  8. Re: vm.kmem_size settings doesn't affect loader?

    On Thu, 2008-09-25 at 16:10 +0100, Pete French wrote:
    > > These are the tuning settings I use:
    > >
    > > vm.kmem_size="1536M"
    > > vm.kmem_size_max="1536M"
    > > vfs.zfs.arc_min="16M"
    > > vfs.zfs.arc_max="64M"

    >
    > >
    > > The entire copying process took almost 2 hours. Not once did I
    > > experience kmem exhaustion. I can *guarantee* that I would have crashed
    > > the box numerous times had I not tuned the machine with the values
    > > above.

    >
    > I am interested in the last two values you have there - I also use ZFS and
    > tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered
    > toi set the arc_min - and the default appears to be 16M anyway. The arc_max
    > value I do set, but I got dreadful performance when I had it at 6M, so
    > I upped it to 128M. This has not resulted in any panics, and gives much better
    > performance. As I understand it there is only one ARC, and it comes from
    > kernel space too, so setting it to an extra 64 meg should not use that much
    > more memory, yes ?

    On my RELENG_7 (circa August 14th)

    vfs.zfs.arc_min: 16777216
    vfs.zfs.arc_max: 402653184

    seem to be set by default. Yes, that is "384M".

    I have

    vm.kmem_size_max="512M"
    vm.kmem_size="512M"

    set in my /boot/loader.conf, and

    vfs.ufs.dirhash_maxmem=16777216
    kern.maxvnodes=100000

    set in /etc/sysctl.conf

    This is amd64 with 3GB of real memory. ZFS pool is used to serve music,
    digital pictures and video to the rest of household, as well as provide
    target for the weekly backups from couple of Windows desktops, so use is
    not that heavy. Machine has been up for 22 days and has shown no ill
    effects that I can notice.

    --
    Alexandre "Sunny" Kovalenko (Олександр Коваленко)

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


+ Reply to Thread