[bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten - Kernel

This is a discussion on [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten - Kernel ; On Mon, Jul 21, 2008 at 01:55:55PM +0200, Ingo Molnar (mingo@elte.hu) wrote: > > > I could try run tests with netconsole deactivated, if you think > > > that's a worthwile line of probing this problem. (although that > ...

+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3
Results 41 to 55 of 55

Thread: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

  1. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    On Mon, Jul 21, 2008 at 01:55:55PM +0200, Ingo Molnar (mingo@elte.hu) wrote:
    > > > I could try run tests with netconsole deactivated, if you think
    > > > that's a worthwile line of probing this problem. (although that
    > > > would make me do blind tests in essence - having kernel log output
    > > > is really essential.)

    > >
    > > Let's try this way first. If system will continue to crash, we will
    > > add some debug options in various pathes. Existing reports do not
    > > contain enough information unfortunately, so we will not lose too
    > > much.

    >
    > ok. I've turned off netconsole - 8 successful bootups in a row so far.
    > The box is a slow booter/builder with an 8 kernels/hour test throughput,
    > so if everything goes fine we should have meaningful results in about 10
    > hours.
    >
    > ( there are other, faster testboxes in -tip testing with 33 kernels/hour
    > build+boot throughput where we'd have to wait only 2 hours - but as
    > per Murphy's law they dont trigger this bug ;-)


    Since 2.6.25 there was only single change in netpoll.c:
    f5184d267c1aedb9b7a8cc44e08ff6b8d382c3b5
    Which looks innocent.

    Is your driver e1000 or e1000e? Can you check different one?

    --
    Evgeniy Polyakov
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten


    * Evgeniy Polyakov wrote:

    > On Mon, Jul 21, 2008 at 01:55:55PM +0200, Ingo Molnar (mingo@elte.hu) wrote:
    > > > > I could try run tests with netconsole deactivated, if you think
    > > > > that's a worthwile line of probing this problem. (although that
    > > > > would make me do blind tests in essence - having kernel log output
    > > > > is really essential.)
    > > >
    > > > Let's try this way first. If system will continue to crash, we will
    > > > add some debug options in various pathes. Existing reports do not
    > > > contain enough information unfortunately, so we will not lose too
    > > > much.

    > >
    > > ok. I've turned off netconsole - 8 successful bootups in a row so far.
    > > The box is a slow booter/builder with an 8 kernels/hour test throughput,
    > > so if everything goes fine we should have meaningful results in about 10
    > > hours.
    > >
    > > ( there are other, faster testboxes in -tip testing with 33 kernels/hour
    > > build+boot throughput where we'd have to wait only 2 hours - but as
    > > per Murphy's law they dont trigger this bug ;-)

    >
    > Since 2.6.25 there was only single change in netpoll.c:
    > f5184d267c1aedb9b7a8cc44e08ff6b8d382c3b5
    > Which looks innocent.
    >
    > Is your driver e1000 or e1000e? Can you check different one?


    i cannot check e1000 anymore due to this upstream commit:

    | d03157babed7424f5391af43200593768ce69c9a is first bad commit
    | commit d03157babed7424f5391af43200593768ce69c9a
    | Author: Auke Kok
    | Date: Sun Jun 22 15:21:29 2008 -0700
    |
    | e1000: remove PCI Express device IDs
    |
    | We do not want to prolong the situation much longer that e1000
    | and e1000e support these devices at the same time. As a result,
    | take out the bandage that was added for the interim period
    | and remove all the PCI Express device IDs from e1000.

    but yes, this box was using e1000 for a long time, and recently migrated
    to e1000e. I'm not sure there's any connection, do you think there is?

    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    Evgeniy Polyakov wrote:
    > Not that obvious...
    > skb->next is cleared in lots of places, in xmit network helper
    > for example, but since rest of the packet was not modified, it
    > means given skb was not freed, so it will not help.
    >
    >


    The skb was definitely freed. 6b is written over an object when it is
    freed. Something else retained a pointer to the skb and was confident
    that the skb still exists.


    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    Vegard Nossum wrote:
    > On Mon, Jul 21, 2008 at 12:50 PM, Ingo Molnar wrote:
    >
    >> the problem is, most of the crashes dont come with any usable dump. This
    >> is a laptop so netconsole is the only reliable route out - and if
    >> something in networking crashes chances are that it hoses netconsole
    >> before it can get anything out.
    >>
    >> Another thing is that i'm activating netconsole on this box via a kernel
    >> boot line and from within a bzImage (to get it activated as early as
    >> possible) - maybe that's a tad too early for certain initialization
    >> sequences?
    >>
    >> I could try run tests with netconsole deactivated, if you think that's a
    >> worthwile line of probing this problem. (although that would make me do
    >> blind tests in essence - having kernel log output is really essential.)
    >>

    >
    > How about posting your custom patches for public review? :-D
    >
    > (There was recently an odd case of kzalloc() succeeding even though it
    > was called before the slab caches were initialized -- and it only
    > generated a warning about irqs-on happening too early. I just mean to
    > say that it _can_ happen. On another machine, it crashed
    > spectacularly. I guess you're not enabling the netconsole before slab
    > allocator is up?)
    >
    >
    > Vegard
    >
    >

    Kzalloc for objects > 4k will succeed even during early boot since we
    will directly call into the page allocator.


    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten


    * Ingo Molnar wrote:

    > * Evgeniy Polyakov wrote:
    >
    > > > I could try run tests with netconsole deactivated, if you think
    > > > that's a worthwile line of probing this problem. (although that
    > > > would make me do blind tests in essence - having kernel log output
    > > > is really essential.)

    > >
    > > Let's try this way first. If system will continue to crash, we will
    > > add some debug options in various pathes. Existing reports do not
    > > contain enough information unfortunately, so we will not lose too
    > > much.

    >
    > ok. I've turned off netconsole - 8 successful bootups in a row so far.
    > The box is a slow booter/builder with an 8 kernels/hour test
    > throughput, so if everything goes fine we should have meaningful
    > results in about 10 hours.
    >
    > ( there are other, faster testboxes in -tip testing with 33 kernels/hour
    > build+boot throughput where we'd have to wait only 2 hours - but as
    > per Murphy's law they dont trigger this bug ;-)


    70 bootups with netconsole turned off, and not a single crash so far.

    So it's now a strong likelyhood that this crash is a combination of
    e1000e+netconsole.

    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten


    * Evgeniy Polyakov wrote:

    > It could be an interesting kmemcheck extension to catch not only
    > non-initialized memory, but also writes into just freed one, but not
    > yet returned by allocator to the next user.


    hm, i think kmemcheck does that mostly already. If not then I think
    kmemcheck should just turn off all buffering that might happen on the
    SLUB level (if it doesnt do it already). I.e. a kfree()d buffer should
    be marked as uninitialized immediately.

    Ingo
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    Hi Christoph.

    On Mon, Jul 21, 2008 at 11:22:27AM -0500, Christoph Lameter (cl@linux-foundation.org) wrote:
    > >Not that obvious...
    > >skb->next is cleared in lots of places, in xmit network helper
    > >for example, but since rest of the packet was not modified, it
    > >means given skb was not freed, so it will not help.

    >
    > The skb was definitely freed. 6b is written over an object when it is
    > freed. Something else retained a pointer to the skb and was confident
    > that the skb still exists.


    I meant it was not tried to be freed second time, since skb->users area
    (the very end of the skb) was not changed from 6b to 6a, but its
    skb->next pointer (first field in the skb) was set to NULL, so after
    that skb was not used at all.

    It could be an interesting kmemcheck extension to catch not only
    non-initialized memory, but also writes into just freed one, but not yet
    returned by allocator to the next user.

    --
    Evgeniy Polyakov
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  8. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    On Mon, Jul 21, 2008 at 6:19 PM, Christoph Lameter
    wrote:
    >> (There was recently an odd case of kzalloc() succeeding even though it
    >> was called before the slab caches were initialized -- and it only
    >> generated a warning about irqs-on happening too early. I just mean to
    >> say that it _can_ happen. On another machine, it crashed
    >> spectacularly. I guess you're not enabling the netconsole before slab
    >> allocator is up?)

    >
    > Kzalloc for objects > 4k will succeed even during early boot since we will
    > directly call into the page allocator.


    It wasn't just that; it got into __slab_alloc() and out again. See the
    backtrace at http://lkml.org/lkml/2008/7/19/23

    It's scary, I know. It's a good thing we have irqtrace :-)


    Vegard

    --
    "The animistic metaphor of the bug that maliciously sneaked in while
    the programmer was not looking is intellectually dishonest as it
    disguises that the error is the programmer's own creation."
    -- E. W. Dijkstra, EWD1036
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  9. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    On Mon, Jul 21, 2008 at 10:05 PM, Ingo Molnar wrote:
    >
    > * Evgeniy Polyakov wrote:
    >
    >> It could be an interesting kmemcheck extension to catch not only
    >> non-initialized memory, but also writes into just freed one, but not
    >> yet returned by allocator to the next user.

    >
    > hm, i think kmemcheck does that mostly already. If not then I think
    > kmemcheck should just turn off all buffering that might happen on the
    > SLUB level (if it doesnt do it already). I.e. a kfree()d buffer should
    > be marked as uninitialized immediately.


    We don't catch writes (yet, because kmemcheck gets confused by SLUB's
    own writes so far), but yes, we mark objects "freed" when they are
    freed (until the slab page is returned to the page allocator) and we
    may detect reads from this area before the object is reallocated. In
    that case, a "read from freed memory" message will appear.

    Unfortunately, it doesn't work quite as well as I'd like it to yet. We
    did catch one real bug in ACPI in this way (read-after-free). But as I
    have discovered, not all machines support single-stepping over REP
    STOS instructions properly. Which means that kmemcheck breaks silently
    on some machines. There's also the recent DMA patch, which might
    improve things. Linus is right, it's really fragile. But we are
    working on it. It is going in the right direction anyway.

    (Pekka and I were also working on an extension to SLUB which delays
    the freeing of objects, i.e. so that objects will remain in the
    "freed" state for a certain amount of time before it can be allocated
    again. But we don't really have a _good_ patch for this yet, it was
    more like an ugly hack at the time.)


    Vegard

    --
    "The animistic metaphor of the bug that maliciously sneaked in while
    the programmer was not looking is intellectually dishonest as it
    disguises that the error is the programmer's own creation."
    -- E. W. Dijkstra, EWD1036
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  10. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    On Mon, Jul 21, 2008 at 09:21:38PM +0200, Ingo Molnar (mingo@elte.hu) wrote:
    > So it's now a strong likelyhood that this crash is a combination of
    > e1000e+netconsole.


    e1000_clean_tx_irq() call looks particulary suspicious: it is called
    without adapter->tx_queue_lock in poll controller (netconsole callback)
    and with that lock in NAPI handler.

    Can you check kind of this patch:

    diff --git a/drivers/net/e1000e/netdev.c b/drivers/net/e1000e/netdev.c
    index 869544b..5f180d8 100644
    --- a/drivers/net/e1000e/netdev.c
    +++ b/drivers/net/e1000e/netdev.c
    @@ -4067,7 +4067,9 @@ static void e1000_netpoll(struct net_device *netdev)
    disable_irq(adapter->pdev->irq);
    e1000_intr(adapter->pdev->irq, netdev);

    + spin_lock(&adapter->tx_queue_lock);
    e1000_clean_tx_irq(adapter);
    + spin_unlock(&adapter->tx_queue_lock);

    enable_irq(adapter->pdev->irq);
    }


    --
    Evgeniy Polyakov
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  11. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    From: Evgeniy Polyakov
    Date: Tue, 22 Jul 2008 01:24:48 +0400

    > On Mon, Jul 21, 2008 at 09:21:38PM +0200, Ingo Molnar (mingo@elte.hu) wrote:
    > > So it's now a strong likelyhood that this crash is a combination of
    > > e1000e+netconsole.

    >
    > e1000_clean_tx_irq() call looks particulary suspicious: it is called
    > without adapter->tx_queue_lock in poll controller (netconsole callback)
    > and with that lock in NAPI handler.
    >
    > Can you check kind of this patch:


    The call even seems pointless, since the caller will call
    ->poll() (which is e1000_clean) as the very next action,
    and that will invoke e1000_clean_tx_irq() properly.

    I would just delete this call from e1000_netpoll() entirely.
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  12. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten


    * David Miller wrote:

    > From: Evgeniy Polyakov
    > Date: Tue, 22 Jul 2008 01:24:48 +0400
    >
    > > On Mon, Jul 21, 2008 at 09:21:38PM +0200, Ingo Molnar (mingo@elte.hu) wrote:
    > > > So it's now a strong likelyhood that this crash is a combination of
    > > > e1000e+netconsole.

    > >
    > > e1000_clean_tx_irq() call looks particulary suspicious: it is called
    > > without adapter->tx_queue_lock in poll controller (netconsole callback)
    > > and with that lock in NAPI handler.
    > >
    > > Can you check kind of this patch:

    >
    > The call even seems pointless, since the caller will call ->poll()
    > (which is e1000_clean) as the very next action, and that will invoke
    > e1000_clean_tx_irq() properly.
    >
    > I would just delete this call from e1000_netpoll() entirely.


    ok, i've added the patch below to tip/out-of-tree.

    Overnight test had about 100 successful bootups on this testbox. (until
    it stopped on a drivers/net/hp.c build error - which is unrelated to
    this problem)

    So testing with netconsole disabled is conclusive enough to implicate
    netconsole strongly. I've now re-enabled netconsole on the testbox and
    will continue the test with the fix below. Previously it would crash
    within 10-40 iterations.

    Ingo

    ----------------->
    commit bf89280dea6d97671aa5f75f2591ae7e8e3e6699
    Author: Ingo Molnar
    Date: Tue Jul 22 09:44:32 2008 +0200

    e1000e: fix e1000_netpoll(), remove extraneous e1000_clean_tx_irq() call

    Evgeniy Polyakov noticed that drivers/net/e1000e/netdev.c:e1000_netpoll()
    was calling e1000_clean_tx_irq() without taking the TX lock.

    David Miller suggested to remove the call altogether: since in this
    callpah there's periodic calls to ->poll() anyway which will do
    e1000_clean_tx_irq() and will garbage-collect any finished TX ring
    descriptors.

    This might solve the e1000e+netconsole crashes i've been seeing:

    ================================================== ===========================
    BUG skbuff_head_cache: Poison overwritten
    -----------------------------------------------------------------------------

    INFO: 0xf658ae9c-0xf658ae9c. First byte 0x6a instead of 0x6b
    INFO: Allocated in __alloc_skb+0x2c/0x110 age=0 cpu=0 pid=5098
    INFO: Freed in __kfree_skb+0x31/0x80 age=0 cpu=1 pid=4440
    INFO: Slab 0xc16cc140 objects=16 used=1 fp=0xf658ae00 flags=0x400000c3
    INFO: Object 0xf658ae00 @offset=3584 fp=0xf658af00

    Signed-off-by: Ingo Molnar
    ---
    drivers/net/e1000e/netdev.c | 2 --
    1 files changed, 0 insertions(+), 2 deletions(-)

    diff --git a/drivers/net/e1000e/netdev.c b/drivers/net/e1000e/netdev.c
    index 869544b..9c0f56b 100644
    --- a/drivers/net/e1000e/netdev.c
    +++ b/drivers/net/e1000e/netdev.c
    @@ -4067,8 +4067,6 @@ static void e1000_netpoll(struct net_device *netdev)
    disable_irq(adapter->pdev->irq);
    e1000_intr(adapter->pdev->irq, netdev);

    - e1000_clean_tx_irq(adapter);
    -
    enable_irq(adapter->pdev->irq);
    }
    #endif
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  13. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten


    * Ingo Molnar wrote:

    > > > Can you check kind of this patch:

    > >
    > > The call even seems pointless, since the caller will call ->poll()
    > > (which is e1000_clean) as the very next action, and that will invoke
    > > e1000_clean_tx_irq() properly.
    > >
    > > I would just delete this call from e1000_netpoll() entirely.

    >
    > ok, i've added the patch below to tip/out-of-tree.
    >
    > Overnight test had about 100 successful bootups on this testbox.
    > (until it stopped on a drivers/net/hp.c build error - which is
    > unrelated to this problem)
    >
    > So testing with netconsole disabled is conclusive enough to implicate
    > netconsole strongly. I've now re-enabled netconsole on the testbox and
    > will continue the test with the fix below. Previously it would crash
    > within 10-40 iterations.


    it's now past 50 successful iterations with e1000e+netconsole enabled
    again with the fix applied, and not a single crash, hang, memory
    corruption or other weirdness happened. Jeff, please apply the fix
    below.

    Thanks,

    Ingo

    ---------------->
    commit 1fd48807674d4f6defe82258c46619ce2c116943
    Author: Ingo Molnar
    Date: Tue Jul 22 09:44:32 2008 +0200

    e1000e: fix e1000_netpoll(), remove extraneous e1000_clean_tx_irq() call

    Evgeniy Polyakov noticed that drivers/net/e1000e/netdev.c:e1000_netpoll()
    was calling e1000_clean_tx_irq() without taking the TX lock.

    David Miller suggested to remove the call altogether: since in this
    callpah there's periodic calls to ->poll() anyway which will do
    e1000_clean_tx_irq() and will garbage-collect any finished TX ring
    descriptors.

    This fix solved the e1000e+netconsole crashes i've been seeing:

    ================================================== ===========================
    BUG skbuff_head_cache: Poison overwritten
    -----------------------------------------------------------------------------

    INFO: 0xf658ae9c-0xf658ae9c. First byte 0x6a instead of 0x6b
    INFO: Allocated in __alloc_skb+0x2c/0x110 age=0 cpu=0 pid=5098
    INFO: Freed in __kfree_skb+0x31/0x80 age=0 cpu=1 pid=4440
    INFO: Slab 0xc16cc140 objects=16 used=1 fp=0xf658ae00 flags=0x400000c3
    INFO: Object 0xf658ae00 @offset=3584 fp=0xf658af00

    Signed-off-by: Ingo Molnar
    ---
    drivers/net/e1000e/netdev.c | 2 --
    1 files changed, 0 insertions(+), 2 deletions(-)

    diff --git a/drivers/net/e1000e/netdev.c b/drivers/net/e1000e/netdev.c
    index 869544b..9c0f56b 100644
    --- a/drivers/net/e1000e/netdev.c
    +++ b/drivers/net/e1000e/netdev.c
    @@ -4067,8 +4067,6 @@ static void e1000_netpoll(struct net_device *netdev)
    disable_irq(adapter->pdev->irq);
    e1000_intr(adapter->pdev->irq, netdev);

    - e1000_clean_tx_irq(adapter);
    -
    enable_irq(adapter->pdev->irq);
    }
    #endif
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  14. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    From: Ingo Molnar
    Date: Tue, 22 Jul 2008 15:34:15 +0200

    > it's now past 50 successful iterations with e1000e+netconsole enabled
    > again with the fix applied, and not a single crash, hang, memory
    > corruption or other weirdness happened. Jeff, please apply the fix
    > below.


    I'll take care of integrating this, thanks Ingo.
    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  15. Re: [bug, netconsole, SLUB] BUG skbuff_head_cache: Poison overwritten

    On Wed, 2008-07-23 at 15:31 -0700, David Miller wrote:
    > From: Ingo Molnar
    > Date: Tue, 22 Jul 2008 15:34:15 +0200
    >
    > > it's now past 50 successful iterations with e1000e+netconsole enabled
    > > again with the fix applied, and not a single crash, hang, memory
    > > corruption or other weirdness happened. Jeff, please apply the fix
    > > below.

    >
    > I'll take care of integrating this, thanks Ingo.
    > --
    > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    > the body of a message to majordomo@vger.kernel.org
    > More majordomo info at http://vger.kernel.org/majordomo-info.html
    > Please read the FAQ at http://www.tux.org/lkml/


    Thanks Dave. I have the patch in my queue of patches to push, just had
    not pushed it to you and Jeff yet.

    --
    Cheers,
    Jeff

    --
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3