On Thursday 25 September 2008 06:26:58 am Bartosz Stec wrote:
>
> Today I've experienced zfs-related kernel panic. Log says:
>
> savecore: reboot after panic: kmem_malloc(131072): kmem_map too
> small: 327684096 total allocated
>
> Reported amount of memory (327684096) is wrong, because i made suggested
> tuning in my loader.conf:
>
> vm.kmem_size="512M"
> vm.kmem_size_max="512M"
>
> Just to be sure:
>
> # sysctl vm | grep kmem
> vm.kmem_size: 536870912
> vm.kmem_size_min: 0
> vm.kmem_size_max: 536870912
> vm.kmem_size_scale: 3
>
> Am I missing something?

If kvm is fragmented you could have a malloc fail even if there are enough
total free bytes for the allocatoin.

--
John Baldwin
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Thu, Sep 25, 2008 at 12:26:58PM +0200, Bartosz Stec wrote:
>
>
> Today I've experienced zfs-related kernel panic. Log says:
>
> savecore: reboot after panic: kmem_malloc(131072): kmem_map too
> small: 327684096 total allocated
>
> Reported amount of memory (327684096) is wrong, because i made suggested
> tuning in my loader.conf:
>
> vm.kmem_size="512M"
> vm.kmem_size_max="512M"
>
> Just to be sure:
>
> # sysctl vm | grep kmem
> vm.kmem_size: 536870912
> vm.kmem_size_min: 0
> vm.kmem_size_max: 536870912
> vm.kmem_size_scale: 3
>
> Am I missing something?

I believe this is normal. The amount shown in "total allocated" will
not match what you have vm.kmem_size or vm.kmem_size_max set to.
Someone more familiar with the VM can explain why this is, but as I
understand it, it's 100% normal.

Your options are:

1) Consider increasing it from 512M to something like 1.5GB; do not
increase it past that on RELENG_7, as there isn't support for more than
2GB total. For example, on a 1GB memory machine, I often recommend
768M. On 2GB machines, 1536M. You will need to run -CURRENT if you
want more.

2) Tune ZFS aggressively. Start by setting vfs.zfs.arc_min="16M"
and vfs.zfs.arc_max="64M".

If your machine has some small amount of memory (768MB, 1GB, etc.),
then you probably shouldn't be using ZFS.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

Jeremy Chadwick pisze:
> On Thu, Sep 25, 2008 at 12:26:58PM +0200, Bartosz Stec wrote:
>
>> Today I've experienced zfs-related kernel panic. Log says:
>>
>> savecore: reboot after panic: kmem_malloc(131072): kmem_map too
>> small: 327684096 total allocated
>>
>> Reported amount of memory (327684096) is wrong, because i made suggested
>> tuning in my loader.conf:
>>
>> vm.kmem_size="512M"
>> vm.kmem_size_max="512M"
>>
>> Just to be sure:
>>
>> # sysctl vm | grep kmem
>> vm.kmem_size: 536870912
>> vm.kmem_size_min: 0
>> vm.kmem_size_max: 536870912
>> vm.kmem_size_scale: 3
>>
>> Am I missing something?
>>
>
> I believe this is normal. The amount shown in "total allocated" will
> not match what you have vm.kmem_size or vm.kmem_size_max set to.
> Someone more familiar with the VM can explain why this is, but as I
> understand it, it's 100% normal.
>
Thanks for explaining. This part of tuning guide confused me:

"I was able to generate the following kernel panic in less than a
minute by copying files from a linux server connected via gigabit
crossover:
Apr 8 06:46:08 nas savecore: reboot after panic: kmem_malloc(131072): kmem_map too small: 528273408 total allocated
As you can see, this is *with* vm.kmem_size="512M"!"

I've just expected similiar numbers in my case.
> Your options are:
>
> 1) Consider increasing it from 512M to something like 1.5GB; do not
> increase it past that on RELENG_7, as there isn't support for more than
> 2GB total. For example, on a 1GB memory machine, I often recommend
> 768M. On 2GB machines, 1536M. You will need to run -CURRENT if you
> want more.
>
> 2) Tune ZFS aggressively. Start by setting vfs.zfs.arc_min="16M"
> and vfs.zfs.arc_max="64M".
>
> If your machine has some small amount of memory (768MB, 1GB, etc.),
> then you probably shouldn't be using ZFS.
>
>
Problem occured on i386 machine with 1GB of memory and 7.1-pre (3HDD,
40GB, RAIDZ1). I know that i386 is highly unrecommended for ZFS, but
it's just a home box for testing and learning purposes - I just want to
know what I'm doing and what should I expect when I decide to put ZFS on
server machines Currently, from posts on freebsd-fs, I conclude that
even with a gigs of kmem and using AMD64, we still can experience panic
from kmem_malloc. Manual tuning is hard for me because I'm not familiar
with BSD kernel code nor kernel memory management. I'm just an end-user
who love concepts of ZFS and wait for it to be (more) stable. Of course
I've followed tuning guide carefully.
So, for now, I will add another 512MB of memory and increse vm.kmem_size
and do more tests. Thanks once again for explaining and suggestions.
Good luck with ZFS everyone!

--
Bartosz Stec

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Thu, Sep 25, 2008 at 04:14:02PM +0200, Bartosz Stec wrote:
>> Your options are:
>>
>> 1) Consider increasing it from 512M to something like 1.5GB; do not
>> increase it past that on RELENG_7, as there isn't support for more than
>> 2GB total. For example, on a 1GB memory machine, I often recommend
>> 768M. On 2GB machines, 1536M. You will need to run -CURRENT if you
>> want more.
>>
>> 2) Tune ZFS aggressively. Start by setting vfs.zfs.arc_min="16M"
>> and vfs.zfs.arc_max="64M".
>>
>> If your machine has some small amount of memory (768MB, 1GB, etc.),
>> then you probably shouldn't be using ZFS.
>>
>>
> Problem occured on i386 machine with 1GB of memory and 7.1-pre (3HDD,
> 40GB, RAIDZ1). I know that i386 is highly unrecommended for ZFS, but
> it's just a home box for testing and learning purposes - I just want to
> know what I'm doing and what should I expect when I decide to put ZFS on
> server machines Currently, from posts on freebsd-fs, I conclude that
> even with a gigs of kmem and using AMD64, we still can experience panic
> from kmem_malloc.

The i386 vs. amd64 argument is bogus, if you ask me. ZFS works on both.
amd64 is recommended because ZFS contains code that makes heavy use of
64-bit values, and because amd64 offers large amounts of addressed
memory without disgusting hacks like PAE.

That said -- yes, even with "gigs of kmem and using AMD64", you can
still panic due to kmem exhaustion. I have fairly decent experience
with this problem, because it haunted me for quite some time.

A large portion of the problem is that kmem_max, on i386 and amd64 (yes,
you read that right) has a 2GB limit on RELENG_7. I repeat: a 2GB
limit, regardless of i386 or amd64.

This limit has been increased to 512GB on CURRENT, but there are no
plans to MFC those changes, as they are too major.

Let me tell you something I did this weekend. I had to copy literally
200GB of data from a ZFS raidz1 pool (spread across 3 disks) to two
different places: 1) a UFS2 filesystem on a different disk, and 2)
across a gigE network to a Windows machine. I had to do this because I
was adding a disk to the vdev, which cannot be done without re-creating
the pool (this is a known problem with ZFS, and has nothing to do with
FreeBSD).

The machine hosting the data runs RELENG_7 with amd64, and contains 4GB
of memory. However, I've accomplished the same task with only 2GB of
memory as well.

These are the tuning settings I use:

vm.kmem_size="1536M"
vm.kmem_size_max="1536M"
vfs.zfs.arc_min="16M"
vfs.zfs.arc_max="64M"

The entire copying process took almost 2 hours. Not once did I
experience kmem exhaustion. I can *guarantee* that I would have crashed
the box numerous times had I not tuned the machine with the values
above.

> Manual tuning is hard for me because I'm not familiar
> with BSD kernel code nor kernel memory management. I'm just an end-user
> who love concepts of ZFS and wait for it to be (more) stable. Of course
> I've followed tuning guide carefully.

I'm an "experienced" end-user who has very little experience with BSD
kernel code and absolutely no experience with kernel memory management.
Proper tuning is all that's needed, regardless of your knowledge set.

Please try installing 2GB of memory in your i386 box, and then use
the exact loader.conf values I specified above.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

> These are the tuning settings I use:
>
> vm.kmem_size="1536M"
> vm.kmem_size_max="1536M"
> vfs.zfs.arc_min="16M"
> vfs.zfs.arc_max="64M"

>
> The entire copying process took almost 2 hours. Not once did I
> experience kmem exhaustion. I can *guarantee* that I would have crashed
> the box numerous times had I not tuned the machine with the values
> above.

I am interested in the last two values you have there - I also use ZFS and
tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered
toi set the arc_min - and the default appears to be 16M anyway. The arc_max
value I do set, but I got dreadful performance when I had it at 6M, so
I upped it to 128M. This has not resulted in any panics, and gives much better
performance. As I understand it there is only one ARC, and it comes from
kernel space too, so setting it to an extra 64 meg should not use that much
more memory, yes ?

My setup is:

vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="128M"
vfs.zfs.prefetch_disable=1

This is on a 2gb amd64 machine, and works beautifully, with no out of
memory errors and panics. I have been trying to panic this for a few
days. I will probably chnage the kmem size to 1536 which is what I use on
other machines though.

-pete.
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Thu, Sep 25, 2008 at 04:10:07PM +0100, Pete French wrote:
> > These are the tuning settings I use:
> >
> > vm.kmem_size="1536M"
> > vm.kmem_size_max="1536M"
> > vfs.zfs.arc_min="16M"
> > vfs.zfs.arc_max="64M"
> >
> > The entire copying process took almost 2 hours. Not once did I
> > experience kmem exhaustion. I can *guarantee* that I would have crashed
> > the box numerous times had I not tuned the machine with the values
> > above.
>
> I am interested in the last two values you have there - I also use ZFS and
> tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered
> toi set the arc_min - and the default appears to be 16M anyway. The arc_max
> value I do set, but I got dreadful performance when I had it at 6M, so
> I upped it to 128M.

ARC tuning is what's necessary to induce ZFS memory constraints. If you
only set kmem_size/max, you will likely run into cases where the ARC
grows to such a size that kmem exhaustion happens. Even the Solaris ZFS
"do not tune ZFS" article outlines this fact (read it all, including the
RFEs):

http://www.solarisinternals.com/wiki..._the_ARC_Cache

Just keep in mind this is specific to Solaris, but the problems
conceptually are identical to FreeBSD.

> This has not resulted in any panics, and gives much better
> performance. As I understand it there is only one ARC, and it comes from
> kernel space too, so setting it to an extra 64 meg should not use that much
> more memory, yes ?

The ARC uses kmem. "Should not use that much more memory" is a matter
of opinion; if an additional 64MB given to ARC causes kmem exhaustion
(there's no global standard for this; keep reading, I explain it near
the bottom of my mail), then decreasing the ARC max is necesary.

On the bright side, it's very easy to determine how much memory is being
dedicated to the ARC. sysctl kstat.zfs.misc will disclose this data
and I believe the following statistics are the ones you have to examine
for this issue:

kstat.zfs.misc.arcstats.p
kstat.zfs.misc.arcstats.c
kstat.zfs.misc.arcstats.c_min
kstat.zfs.misc.arcstats.c_max
kstat.zfs.misc.arcstats.size

On my system (the one using the tuning parameters you're questioning):

kstat.zfs.misc.arcstats.p: 60487680
kstat.zfs.misc.arcstats.c: 67108864
kstat.zfs.misc.arcstats.c_min: 16777216
kstat.zfs.misc.arcstats.c_max: 67108864
kstat.zfs.misc.arcstats.size: 67092480

For those wanting to read about the ARC and what it's for:

http://en.wikipedia.org/wiki/Adaptive_Replacement_Cache

> My setup is:
>
> vm.kmem_size="1024M"
> vm.kmem_size_max="1024M"
> vfs.zfs.arc_max="128M"
> vfs.zfs.prefetch_disable=1
>
> This is on a 2gb amd64 machine, and works beautifully, with no out of
> memory errors and panics. I have been trying to panic this for a few
> days. I will probably chnage the kmem size to 1536 which is what I use on
> other machines though.

Disabling prefetch shouldn't have any effect on stability -- many of us
(including FreeBSD core members) have found that disabling prefetch
increases system responsiveness during heavy I/O, and results in a less
"bursty" system. I disable prefetch, but it should have no bearing on
the issue we're discussing.

I'm sure the memory tuning values work fine for you, but they do not for
me; I can absolutely panic the machine with a kmem size of 1GB. How?
Because it's entirely dependent upon the type of ZFS pools/setup a
person has, how much actual load they put on ZFS, and probably a hundred
other things.

Let's adhere to the K.I.S.S. concept, and not turn this thread into
a "tuning micro-management" thread.

What end-users want is ZFS that "just works", which on Solaris 10 it
does. But I have no qualms with tuning ZFS on FreeBSD, though others
seem to. A "tuning guide" cannot easily be written because there are
too many factors to take into consideration. Besides, such a guide
(documenting "how it all interconnects, memory-wise") would end up being
tailored to the technical, which won't help end-users. The existing
"ZFS tuning guide" for FreeBSD keeps things vague for a good reason --
I'm absolutely 100% positive an extensive document would just confuse
the living daylights out of people.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Thu, 2008-09-25 at 16:10 +0100, Pete French wrote:
> > These are the tuning settings I use:
> >
> > vm.kmem_size="1536M"
> > vm.kmem_size_max="1536M"
> > vfs.zfs.arc_min="16M"
> > vfs.zfs.arc_max="64M"
>
> >
> > The entire copying process took almost 2 hours. Not once did I
> > experience kmem exhaustion. I can *guarantee* that I would have crashed
> > the box numerous times had I not tuned the machine with the values
> > above.
>
> I am interested in the last two values you have there - I also use ZFS and
> tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered
> toi set the arc_min - and the default appears to be 16M anyway. The arc_max
> value I do set, but I got dreadful performance when I had it at 6M, so
> I upped it to 128M. This has not resulted in any panics, and gives much better
> performance. As I understand it there is only one ARC, and it comes from
> kernel space too, so setting it to an extra 64 meg should not use that much
> more memory, yes ?
On my RELENG_7 (circa August 14th)

vfs.zfs.arc_min: 16777216
vfs.zfs.arc_max: 402653184

seem to be set by default. Yes, that is "384M".

I have

vm.kmem_size_max="512M"
vm.kmem_size="512M"

set in my /boot/loader.conf, and

vfs.ufs.dirhash_maxmem=16777216
kern.maxvnodes=100000

set in /etc/sysctl.conf

This is amd64 with 3GB of real memory. ZFS pool is used to serve music,
digital pictures and video to the rest of household, as well as provide
target for the weekly backups from couple of Windows desktops, so use is
not that heavy. Machine has been up for 22 days and has shown no ill
effects that I can notice.

--
Alexandre "Sunny" Kovalenko (Олександр Коваленко)

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"