[bug] block subsystem related crash with latest -git - Kernel

This is a discussion on [bug] block subsystem related crash with latest -git - Kernel ; * Jens Axboe wrote: > On Wed, Oct 17 2007, Ingo Molnar wrote: > > > > btw., i get the following build warning: > > > > drivers/scsi/scsi_lib.c: In function 'scsi_free_sgtable': > > drivers/scsi/scsi_lib.c:708: warning: 'index' may be used ...

+ Reply to Thread
Page 3 of 8 FirstFirst 1 2 3 4 5 ... LastLast
Results 41 to 60 of 151

Thread: [bug] block subsystem related crash with latest -git

  1. Re: [bug] ata subsystem related crash with latest -git


    * Jens Axboe wrote:

    > On Wed, Oct 17 2007, Ingo Molnar wrote:
    > >
    > > btw., i get the following build warning:
    > >
    > > drivers/scsi/scsi_lib.c: In function 'scsi_free_sgtable':
    > > drivers/scsi/scsi_lib.c:708: warning: 'index' may be used uninitialized in this function
    > > drivers/scsi/scsi_lib.c:708: note: 'index' was declared here
    > > drivers/scsi/scsi_lib.c:708: warning: 'index' may be used uninitialized in this function
    > > drivers/scsi/scsi_lib.c:708: note: 'index' was declared here
    > > drivers/scsi/scsi_lib.c: In function 'scsi_alloc_sgtable':
    > > drivers/scsi/scsi_lib.c:708: warning: 'index' may be used uninitialized in this function
    > > drivers/scsi/scsi_lib.c:708: note: 'index' was declared here
    > >
    > > not sure it matters.

    >
    > Hmm, I don't see them here (gcc 4.2.1). Must be the BUG(), does it go
    > away if you add a index = -1 in the default: case? Just to be
    > absolutely sure...


    well if i initialize the variable then of course the warning goes away


    NOTE: i get the same warning even without the BUG_ON() patch, and
    without your other fix patch as well. I'm using gcc 4.2.2. (Do you get
    the warning if you pick up the config i sent you earlier today?)

    Ingo
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. Re: [bug] ata subsystem related crash with latest -git

    On Wed, Oct 17 2007, Ingo Molnar wrote:
    >
    > * Ingo Molnar wrote:
    >
    > > > > drivers/scsi/scsi_lib.c:708: warning: 'index' may be used uninitialized in this function
    > > > > drivers/scsi/scsi_lib.c:708: note: 'index' was declared here
    > > > >
    > > > > not sure it matters.
    > > >
    > > > Hmm, I don't see them here (gcc 4.2.1). Must be the BUG(), does it go
    > > > away if you add a index = -1 in the default: case? Just to be
    > > > absolutely sure...

    > >
    > > well if i initialize the variable then of course the warning goes away
    > >
    > >
    > > NOTE: i get the same warning even without the BUG_ON() patch, and
    > > without your other fix patch as well. I'm using gcc 4.2.2. (Do you get
    > > the warning if you pick up the config i sent you earlier today?)

    >
    > btw., i changed the initialization to -1 and the crash still occurs
    > (as expected).


    Would think so, thanks for checking though.

    --
    Jens Axboe

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: [bug] ata subsystem related crash with latest -git

    On Wed, Oct 17 2007, Ingo Molnar wrote:
    >
    > * Jens Axboe wrote:
    >
    > > On Wed, Oct 17 2007, Ingo Molnar wrote:
    > > >
    > > > btw., i get the following build warning:
    > > >
    > > > drivers/scsi/scsi_lib.c: In function 'scsi_free_sgtable':
    > > > drivers/scsi/scsi_lib.c:708: warning: 'index' may be used uninitialized in this function
    > > > drivers/scsi/scsi_lib.c:708: note: 'index' was declared here
    > > > drivers/scsi/scsi_lib.c:708: warning: 'index' may be used uninitialized in this function
    > > > drivers/scsi/scsi_lib.c:708: note: 'index' was declared here
    > > > drivers/scsi/scsi_lib.c: In function 'scsi_alloc_sgtable':
    > > > drivers/scsi/scsi_lib.c:708: warning: 'index' may be used uninitialized in this function
    > > > drivers/scsi/scsi_lib.c:708: note: 'index' was declared here
    > > >
    > > > not sure it matters.

    > >
    > > Hmm, I don't see them here (gcc 4.2.1). Must be the BUG(), does it go
    > > away if you add a index = -1 in the default: case? Just to be
    > > absolutely sure...

    >
    > well if i initialize the variable then of course the warning goes away
    >


    Just making 100% sure that was the missing place, I try not to take
    anything for granted with crazy bugs like this :-)
    >
    > NOTE: i get the same warning even without the BUG_ON() patch, and
    > without your other fix patch as well. I'm using gcc 4.2.2. (Do you get
    > the warning if you pick up the config i sent you earlier today?)


    Let me check... Yep, I do!

    --
    Jens Axboe

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  4. Re: [bug] ata subsystem related crash with latest -git



    On Wed, 17 Oct 2007, Ingo Molnar wrote:
    >
    > nope, this did not help. First bootup went fine, second bootup crashed
    > again (see below), without hitting the BUG_ON().


    I think you'll always hit it if you have a scatter-gather list that is
    exactly filled up.

    Why? Because those things do "sg_next()" on the last entry, and as
    mentioned, that ends up actually accessing one past the end - even if the
    end result is not actually ever *used* (because we just effectively
    incremented it to past the last entry when the code was done with the SG
    list).

    So I think the sg_next() interface is fundamentally mis-designed. It
    should do the scatter-gather link following on *starting* to use the SG
    entry, not after finishing with it.

    Put another way: I suspect pretty much every single sg_next() out there is
    likely to hit this issue. The way that blk_rq_map_sg() fixed its problem
    was exactly to move the "sg_next()" to *before* the use of the SG (and
    even that one is somewhat bogus, in that it just blindly assumes that the
    first entry is not a link entry).

    I suspect the "the next entry is a link" bit should be in the *previous*
    entry, and then sg_next() could look like

    if (next_is_link(sg))
    sg = sg_chain_ptr(sg+1);
    else
    sg++;
    return sg;

    and that would work.

    The alternative is to always make sure to allocate one more SG entry than
    required, so that the last entry is always either the link, or an unused
    entry!

    Linus
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: [bug] ata subsystem related crash with latest -git

    On Wed, Oct 17 2007, Linus Torvalds wrote:
    >
    >
    > On Wed, 17 Oct 2007, Ingo Molnar wrote:
    > >
    > > nope, this did not help. First bootup went fine, second bootup crashed
    > > again (see below), without hitting the BUG_ON().

    >
    > I think you'll always hit it if you have a scatter-gather list that is
    > exactly filled up.
    >
    > Why? Because those things do "sg_next()" on the last entry, and as
    > mentioned, that ends up actually accessing one past the end - even if the
    > end result is not actually ever *used* (because we just effectively
    > incremented it to past the last entry when the code was done with the SG
    > list).
    >
    > So I think the sg_next() interface is fundamentally mis-designed. It
    > should do the scatter-gather link following on *starting* to use the SG
    > entry, not after finishing with it.
    >
    > Put another way: I suspect pretty much every single sg_next() out there is
    > likely to hit this issue. The way that blk_rq_map_sg() fixed its problem
    > was exactly to move the "sg_next()" to *before* the use of the SG (and
    > even that one is somewhat bogus, in that it just blindly assumes that the
    > first entry is not a link entry).
    >
    > I suspect the "the next entry is a link" bit should be in the *previous*
    > entry, and then sg_next() could look like
    >
    > if (next_is_link(sg))
    > sg = sg_chain_ptr(sg+1);
    > else
    > sg++;
    > return sg;
    >
    > and that would work.
    >
    > The alternative is to always make sure to allocate one more SG entry than
    > required, so that the last entry is always either the link, or an unused
    > entry!


    OK, I think you have a very good point here. Ingo, can you verify it
    goes away if apply the below? I have to tend to Other Life stuff but
    will take this up again tomorrow morning and fix the sg_next() usage.

    Linus, please still pull the branch I asked you to earlier. Thanks!


    diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
    index aac8a02..58ede7e 100644
    --- a/drivers/scsi/scsi_lib.c
    +++ b/drivers/scsi/scsi_lib.c
    @@ -39,7 +39,7 @@
    * (unless chaining is used). Should ideally fit inside a single page, to
    * avoid a higher order allocation.
    */
    -#define SCSI_MAX_SG_SEGMENTS 128
    +#define SCSI_MAX_SG_SEGMENTS 129

    struct scsi_host_sg_pool {
    size_t size;


    --
    Jens Axboe

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: [bug] ata subsystem related crash with latest -git



    On Wed, 17 Oct 2007, Jens Axboe wrote:
    >
    > OK, I think you have a very good point here. Ingo, can you verify it
    > goes away if apply the below? I have to tend to Other Life stuff but
    > will take this up again tomorrow morning and fix the sg_next() usage.


    This one will only fix the ones that use the SCSI-lib routines, so it will
    leave everything else broken, no?

    > Linus, please still pull the branch I asked you to earlier. Thanks!


    Already did.

    Linus
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. Re: [bug] ata subsystem related crash with latest -git



    On Wed, 17 Oct 2007, Linus Torvalds wrote:
    >
    > I think you'll always hit it if you have a scatter-gather list that is
    > exactly filled up.


    In fact, I think you'll hit it even if you use the "for_each_sg()"
    helper function. Exactly because it does the sg = sg_next(sg) at the end
    of the for-loop, so it will do it for the last entry too, even though that
    will access one past the end.

    So it really *is* the case that every *single* use of that SG chain needs
    to be switched over to only do the sg_next() when required, or sg_next()
    needs to be fixed to look at the current-in-use entry (which we *can*
    access) when it decides what to do about the next one (which we can *not*
    access).

    Moving the sg_is_chain() bit to the previous entry would work, but it
    requires that nobody who could have a chained scatterlist must *ever*
    access "sg->page" directly - you'd always need to use some helper function
    that masks off the bit, eg

    - rename "sg->page" into "sh->page_and_flag", and make it "unsigned long"
    instead of a pointer.

    - change every single "sg->page" access to use "sg_page(sg)" instead:

    #define sg_pointer(sg) ((void *)((sg)->page_and_flag & ~1ul))

    static inline struct page *sg_page(struct scatterist *sg)
    {
    return sg_pointer(sg);
    }

    - change "sg_next()" to do

    static inline struct scatterlist *sg_next(struct scatterlist *sg)
    {
    if (sg->page_and_flag & 1)
    sg = sg_pointer(sg+1)-1;
    return ++sg;
    }

    where the magic is exactly the fact that now "sg_next()" will *not*
    derefence the next SG entry unless the current one was marked
    appropriately.

    And then *creating* the chain obviously needs to properly mark the
    next-to-last entry with the "next entry is a pointer" flag.

    Big diff, it sounds like. But I don't see many alternatives. Jens?

    Linus
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  8. Re: [bug] ata subsystem related crash with latest -git

    On Wed, Oct 17 2007, Linus Torvalds wrote:
    >
    >
    > On Wed, 17 Oct 2007, Linus Torvalds wrote:
    > >
    > > I think you'll always hit it if you have a scatter-gather list that is
    > > exactly filled up.

    >
    > In fact, I think you'll hit it even if you use the "for_each_sg()"
    > helper function. Exactly because it does the sg = sg_next(sg) at the end
    > of the for-loop, so it will do it for the last entry too, even though that
    > will access one past the end.
    >
    > So it really *is* the case that every *single* use of that SG chain needs
    > to be switched over to only do the sg_next() when required, or sg_next()
    > needs to be fixed to look at the current-in-use entry (which we *can*
    > access) when it decides what to do about the next one (which we can *not*
    > access).


    Auch yes, ok nevermind that last email. There really is no way around
    allocating an extra 'pad' entry right now, so the SCSI_MAX_SG_SEGMENTS
    at 129 should fix Ingo's oops and the other crap is void.

    > Moving the sg_is_chain() bit to the previous entry would work, but it
    > requires that nobody who could have a chained scatterlist must *ever*
    > access "sg->page" directly - you'd always need to use some helper function
    > that masks off the bit, eg
    >
    > - rename "sg->page" into "sh->page_and_flag", and make it "unsigned long"
    > instead of a pointer.
    >
    > - change every single "sg->page" access to use "sg_page(sg)" instead:
    >
    > #define sg_pointer(sg) ((void *)((sg)->page_and_flag & ~1ul))
    >
    > static inline struct page *sg_page(struct scatterist *sg)
    > {
    > return sg_pointer(sg);
    > }
    >
    > - change "sg_next()" to do
    >
    > static inline struct scatterlist *sg_next(struct scatterlist *sg)
    > {
    > if (sg->page_and_flag & 1)
    > sg = sg_pointer(sg+1)-1;
    > return ++sg;
    > }
    >
    > where the magic is exactly the fact that now "sg_next()" will *not*
    > derefence the next SG entry unless the current one was marked
    > appropriately.
    >
    > And then *creating* the chain obviously needs to properly mark the
    > next-to-last entry with the "next entry is a pointer" flag.
    >
    > Big diff, it sounds like. But I don't see many alternatives. Jens?


    Big diff is not a problem for me, my primary concern would be making
    sure that the interface is solid. The above also has the nice side
    effect of making all sg elements usable, so we don't waste any. IIRC
    this was even debated months ago when I first proposed sg chaining, I
    shied away from it because I thought it would seem huge and too
    invasive. Ah well. I'll get digging on this.

    --
    Jens Axboe

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  9. Re: [bug] ata subsystem related crash with latest -git

    On Wed, Oct 17 2007, Linus Torvalds wrote:
    >
    >
    > On Wed, 17 Oct 2007, Jens Axboe wrote:
    > >
    > > OK, I think you have a very good point here. Ingo, can you verify it
    > > goes away if apply the below? I have to tend to Other Life stuff but
    > > will take this up again tomorrow morning and fix the sg_next() usage.

    >
    > This one will only fix the ones that use the SCSI-lib routines, so it will
    > leave everything else broken, no?


    Yes, it'll still crap out if you use debug page alloc for anything else
    :/

    > > Linus, please still pull the branch I asked you to earlier. Thanks!

    >
    > Already did.


    Thanks

    --
    Jens Axboe

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  10. Re: [bug] ata subsystem related crash with latest -git

    On Wed, Oct 17 2007, Jens Axboe wrote:
    > On Wed, Oct 17 2007, Linus Torvalds wrote:
    > >
    > >
    > > On Wed, 17 Oct 2007, Ingo Molnar wrote:
    > > >
    > > > nope, this did not help. First bootup went fine, second bootup crashed
    > > > again (see below), without hitting the BUG_ON().

    > >
    > > I think you'll always hit it if you have a scatter-gather list that is
    > > exactly filled up.
    > >
    > > Why? Because those things do "sg_next()" on the last entry, and as
    > > mentioned, that ends up actually accessing one past the end - even if the
    > > end result is not actually ever *used* (because we just effectively
    > > incremented it to past the last entry when the code was done with the SG
    > > list).


    Well, hang on - where does it end up doing sg_next() on the LAST sg
    entry? I'd argue that this is a bug, like it was in ll_rw_blk.c. I still
    agree that I should make the interface more robust, I just don't see
    where libata ends up doing the sg_next() on the last entry.

    I'm assuming that Ingo is using the previous patches, so that
    blk_rq_map_sg() is using this construct:

    new_segment:
    if (!sg)
    sg = sglist;
    else
    sg = sg_next(sg);

    and the memset() in scsi_alloc_sgtable(), which I'm pretty sure he is.
    I'm assuming we're not hitting pio paths, so there are no manual
    sg_next() calls. Ingo, if you care, can you test this one as well?

    No rush, as mentioned I'll be back tomorrow morning...

    diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
    index bbaa545..0246b61 100644
    --- a/drivers/ata/libata-core.c
    +++ b/drivers/ata/libata-core.c
    @@ -4664,7 +4664,7 @@ static int ata_sg_setup(struct ata_queued_cmd *qc)
    {
    struct ata_port *ap = qc->ap;
    struct scatterlist *sg = qc->__sg;
    - struct scatterlist *lsg = sg_last(qc->__sg, qc->n_elem);
    + struct scatterlist *lsg = &qc->__sg[qc->n_elem - 1];
    int n_elem, pre_n_elem, dir, trim_sg = 0;

    VPRINTK("ENTER, ata%u\n", ap->print_id);

    --
    Jens Axboe

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  11. Re: [bug] ata subsystem related crash with latest -git


    * Jens Axboe wrote:

    > Ingo, if you care, can you test this one as well?


    hm, it still crashes - find the log below.

    Ingo

    [ 34.653682] EXT3-fs: mounted filesystem with ordered data mode.
    [ 34.659487] VFS: Mounted root (ext3 filesystem) readonly.
    [ 34.665208] Freeing unused kernel memory: 372k freed
    [ 35.809988] BUG: unable to handle kernel paging request at virtual address 7c8d3000
    [ 35.817483] printing eip: 784e9480 *pde = 00dda027 *pte = 048d3000
    [ 35.823723] Oops: 0000 [#1] DEBUG_PAGEALLOC
    [ 35.827883]
    [ 35.829357] Pid: 47, comm: rc.sysinit Not tainted (2.6.23 #14)
    [ 35.835162] EIP: 0060:[<784e9480>] EFLAGS: 00010006 CPU: 0
    [ 35.840626] EIP is at ata_qc_issue+0xd0/0x340
    [ 35.844954] EAX: 3fd79000 EBX: 7c8d3000 ECX: 00000020 EDX: 00000020
    [ 35.851194] ESI: 7c8a1a80 EDI: 7c8d2e00 EBP: 7b54007c ESP: 7c8b3d5c
    [ 35.857434] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
    [ 35.862807] Process rc.sysinit (pid: 47, ti=7c8b2000 task=7c86d000 task.ti=7c8b2000)
    [ 35.870346] Stack: 7c8d3000 7b540000 7b540000 00000000 7c8d2ff0 7b54007c 7c8a1a80 7b5417a4
    [ 35.878665] 784c2490 784ef61e 784f2173 7b52de98 7c8a1a80 7b540000 7b5417a4 7c8a1a80
    [ 35.886985] 7b540000 7b524004 784f2260 784ef300 784c2490 7c8a1a80 00000216 7b524004
    [ 35.895304] Call Trace:
    [ 35.897905] [<784c2490>] scsi_done+0x0/0x20
    [ 35.902151] [<784ef61e>] ata_scsi_translate+0xbe/0x140
    [ 35.907350] [<784f2173>] ata_scsi_queuecmd+0x33/0x200
    [ 35.912463] [<784f2260>] ata_scsi_queuecmd+0x120/0x200
    [ 35.917663] [<784ef300>] ata_scsi_rw_xlat+0x0/0x220
    [ 35.922602] [<784c2490>] scsi_done+0x0/0x20
    [ 35.926849] [<784c2d12>] scsi_dispatch_cmd+0x152/0x290
    [ 35.932049] [<78135c9c>] trace_hardirqs_on+0x9c/0xb0
    [ 35.937075] [<784c8c3e>] scsi_request_fn+0x1be/0x370
    [ 35.942102] [<78120f02>] del_timer+0x62/0x70
    [ 35.946434] [<784072d5>] __generic_unplug_device+0x25/0x30
    [ 35.951981] [<784075a5>] generic_unplug_device+0x15/0x30
    [ 35.957354] [<78404e00>] blk_backing_dev_unplug+0x0/0x10
    [ 35.962726] [<78404e0c>] blk_backing_dev_unplug+0xc/0x10
    [ 35.968100] [<78180a9d>] block_sync_page+0x2d/0x40
    [ 35.972952] [<78144dc9>] sync_page+0x29/0x40
    [ 35.977286] [<7876f5dc>] __wait_on_bit_lock+0x3c/0x70
    [ 35.982399] [<78144da0>] sync_page+0x0/0x40
    [ 35.986645] [<78144d82>] __lock_page+0x52/0x60
    [ 35.991152] [<7812adb0>] wake_bit_function+0x0/0x60
    [ 35.996092] [<78144f60>] find_lock_page+0x60/0xa0
    [ 36.000858] [<781473b9>] filemap_fault+0x269/0x3b0
    [ 36.005711] [<78150b7f>] __do_fault+0x4f/0x3e0
    [ 36.010217] [<78147150>] filemap_fault+0x0/0x3b0
    [ 36.014897] [<78152640>] handle_mm_fault+0xf0/0x6c0
    [ 36.019837] [<7810fd5c>] do_page_fault+0x21c/0x670
    [ 36.024689] [<7812e6fd>] down_read_trylock+0x4d/0x60
    [ 36.029716] [<7810fdbb>] do_page_fault+0x27b/0x670
    [ 36.034568] [<7810fb40>] do_page_fault+0x0/0x670
    [ 36.039248] [<787710ea>] error_code+0x6a/0x70
    [ 36.043669] =======================
    [ 36.047221] Code: 84 d9 01 00 00 7e 32 31 d2 89 f6 8b 1c 24 83 c2 01 8b 03 2b 05 f8 ec d7 78 c1 f8 05 c1 e0 0c 03 43 04 89 43 08 83 c3 10 89 1c 24 <8b> 03 a8 01 0f 85 58 02 00 00 39 ca 75 d2 f0 83 44 24 00 00 85
    [ 36.066027] EIP: [<784e9480>] ata_qc_issue+0xd0/0x340 SS:ESP 0068:7c8b3d5c
    [ 100.077701] BUG: spinlock lockup on CPU#0, scsi_eh_0/32, 7b52de98
    [ 100.083667] [<78414b9b>] _raw_spin_lock+0x13b/0x140
    [ 100.088605] [<784c6610>] scsi_error_handler+0x0/0x4f0
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  12. Re: [bug] ata subsystem related crash with latest -git

    On Wed, Oct 17 2007, Jens Axboe wrote:
    > On Wed, Oct 17 2007, Linus Torvalds wrote:
    > >
    > >
    > > On Wed, 17 Oct 2007, Jens Axboe wrote:
    > > >
    > > > OK, I think you have a very good point here. Ingo, can you verify it
    > > > goes away if apply the below? I have to tend to Other Life stuff but
    > > > will take this up again tomorrow morning and fix the sg_next() usage.

    > >
    > > This one will only fix the ones that use the SCSI-lib routines, so it will
    > > leave everything else broken, no?

    >
    > Yes, it'll still crap out if you use debug page alloc for anything else
    > :/


    So until that is fixed up, how about this? Should cover all block
    devices, and I don't think sg_next()/for_each_sg() has spread outside of
    that yet.

    diff --git a/block/ll_rw_blk.c b/block/ll_rw_blk.c
    index 3935469..8708ad0 100644
    --- a/block/ll_rw_blk.c
    +++ b/block/ll_rw_blk.c
    @@ -631,7 +631,7 @@ void blk_queue_max_phys_segments(struct request_queue *q,
    printk("%s: set to minimum %d\n", __FUNCTION__, max_segments);
    }

    - q->max_phys_segments = max_segments;
    + q->max_phys_segments = max_segments - 1;
    }

    EXPORT_SYMBOL(blk_queue_max_phys_segments);

    --
    Jens Axboe

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  13. Re: [bug] ata subsystem related crash with latest -git



    On Wed, 17 Oct 2007, Jens Axboe wrote:
    >
    > Well, hang on - where does it end up doing sg_next() on the LAST sg
    > entry?


    They pretty much *all* do, as far as I can tell.

    For example, to look at the very first one:

    #define for_each_sg(sglist, sg, nr, __i) \
    for (__i = 0, sg = (sglist); __i < (nr); __i++, sg = sg_next(sg))

    let's say that "nr" is 1 (and that's also the allocation size), and look
    at what that causes.

    Right. It causes us to do "sg = sg_next(sg)" once. Which will do what? It
    will increment sg (so that it now points past the single-entry array!) and
    then it will dereference that invalid entry to see if it's a chain entry!


    And no, "1" is not the special case. The special case is *any* time when
    you iterate over as many sg entries as you allocated. You *always* have to
    leave the last one unused in order to avoid this "access past the end"
    problem.

    The alternative is to rewrite the loop, but it's going to be ugly as hell,
    and you need to do that for *every*single*user* of sg_next(). You can
    re-write the above loop as something like

    define for_each_sg(sglist, sg, nr, __i)
    for (__i = 0, sg = NULL ;
    __i < (nr) && sg = (sg ? sglist : sg_next(sg) ;
    __i++))

    but the important part here is that you must do that "sg_next()" *after*
    you have broken out of the loop, and you must basically do it one less
    time than you go through the loop.

    IOW, any loop that leaves "sg" pointing to past the array is inevitably
    buggy, because it will have accessed that last past-tne-end entry as part
    of tryign to decide whether it should perhaps follow a link.

    > I'd argue that this is a bug, like it was in ll_rw_blk.c. I still
    > agree that I should make the interface more robust, I just don't see
    > where libata ends up doing the sg_next() on the last entry.


    See above. I think the exact sequence may be:

    ata_qc_issue()
    (implicitly inlined) ata_sg_setup()
    (explicitly inlined) dma_map_sg()
    (macro) for_each_sg()

    but I didn't see if there are other possible chains that get you to one of
    those invalid sg loops.

    And no, it's *not* just for_each_sg(). Pretty much any "natural" loop over
    the SG list will cause it, because that's how you write loops in C: you
    almost always end up pointing to one past the last entry after the loop.

    Linus
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  14. Re: [bug] block subsystem related crash with latest -git

    Il Wed, Oct 17, 2007 at 07:29:32PM +0200, Jens Axboe ha scritto:
    > OK, it is fine, as long as the sglist is cleared initially. And I don't
    > think there's anyway around that, clearly I didn't think long enough
    > before including the memset() removal from Tomo.
    >
    > Ingo, please try this rolled up version.
    >
    > Linus, this should work. It would probably be best if you first did a
    > git revert on f5c0dde4c66421a3a2d7d6fa604a712c9b0744e5 and then applied
    > the ll_rw_blk.c bit alone. Do you want me to stuff that (revert + patch)
    > into a branch for you to pull?
    >
    > diff --git a/block/ll_rw_blk.c b/block/ll_rw_blk.c
    > index 9e3f3cc..3935469 100644
    > --- a/block/ll_rw_blk.c
    > +++ b/block/ll_rw_blk.c
    > @@ -1322,8 +1322,8 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq,
    > struct scatterlist *sglist)
    > {
    > struct bio_vec *bvec, *bvprv;
    > - struct scatterlist *next_sg, *sg;
    > struct req_iterator iter;
    > + struct scatterlist *sg;
    > int nsegs, cluster;
    >
    > nsegs = 0;
    > @@ -1333,7 +1333,7 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq,
    > * for each bio in rq
    > */
    > bvprv = NULL;
    > - sg = next_sg = &sglist[0];
    > + sg = NULL;
    > rq_for_each_segment(bvec, rq, iter) {
    > int nbytes = bvec->bv_len;
    >
    > @@ -1349,8 +1349,10 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq,
    > sg->length += nbytes;
    > } else {
    > new_segment:
    > - sg = next_sg;
    > - next_sg = sg_next(sg);
    > + if (!sg)
    > + sg = sglist;
    > + else
    > + sg = sg_next(sg);
    >
    > memset(sg, 0, sizeof(*sg));
    > sg->page = bvec->bv_page;
    > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
    > index 0c86be7..aac8a02 100644
    > --- a/drivers/scsi/scsi_lib.c
    > +++ b/drivers/scsi/scsi_lib.c
    > @@ -764,6 +764,8 @@ struct scatterlist *scsi_alloc_sgtable(struct scsi_cmnd *cmd, gfp_t gfp_mask)
    > if (unlikely(!sgl))
    > goto enomem;
    >
    > + memset(sgl, 0, sizeof(*sgl) * sgp->size);
    > +
    > /*
    > * first loop through, set initial index and return value
    > */
    >


    Hi Jens,
    I just hit what I believe is the same bug, though with a slightly
    different OOPS (I see a NULL pointer dereference, see below). The
    patches fixes the bug, and - as Linus suggested - the piece touching
    scsi_lib.c is enough.

    The OOPS:

    scsi4 : ata_piix
    scsi5 : ata_piix
    ata5: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xffa0 irq 14
    ata6: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xffa8 irq 15
    sda1 sda2 sda3 < sda5 sda6 >
    sd 0:0:0:0: [sda] Attached SCSI disk
    ata5.00: ATAPI: MAT****ADVD-RAM UJ-850S, 1.22, max UDMA/33
    Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP:
    [] blk_rq_map_sg+0xf6/0x16a
    PGD 4711067 PUD 508c067 PMD 0
    Oops: 0000 [1] PREEMPT SMP
    CPU 0
    Modules linked in: sd_mod ohci1394 ehci_hcd ata_piix ieee1394 ahci uhci_hcd libata scsi_mod usbcore thermal processor fan
    Pid: 0, comm: swapper Not tainted 2.6.23-ge6d5a11d #3
    RIP: 0010:[] [] blk_rq_map_sg+0xf6/0x16a
    RSP: 0018:ffffffff804eccd8 EFLAGS: 00010087
    RAX: 0000000005331000 RBX: ffff810004740220 RCX: ffff810001000000
    RDX: 0000000000000000 RSI: ffff8100052d4f50 RDI: 0000000005142c00
    RBP: 0000000000001400 R08: 0000000000000000 R09: ffff8100052a3980
    R10: 0000000005143000 R11: 0000000000000400 R12: ffff8100047a4000
    R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
    FS: 0000000000000000(0000) GS:ffffffff8049a000(0000) knlGS:0000000000000000
    CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
    CR2: 0000000000000020 CR3: 0000000005118000 CR4: 00000000000006e0
    DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    Process swapper (pid: 0, threadinfo ffffffff804ac000, task ffffffff8046c340)
    Stack: ffffffff88041834 000000010314ba00 ffff81000477fa80 ffff81000503dd80
    ffff81000503dd80 ffff81000517fc00 000000000000001e 0000000000001d4c
    ffffffff8804193c 0000000015925849 ffff81000473f800 ffff81000503dd80
    Call Trace:
    [] :scsi_mod:scsi_alloc_sgtable+0xad/0x13c
    [] :scsi_mod:scsi_init_io+0x79/0xd9
    [] :sd_mod:sd_prep_fn+0x59/0x367
    [] elv_next_request+0xc0/0x14a
    [] :scsi_mod:scsi_request_fn+0x88/0x356
    [] blk_run_queue+0x41/0x72
    [] :scsi_mod:scsi_next_command+0x2d/0x39
    [] :scsi_mod:scsi_end_request+0xbb/0xc9
    [] :scsi_mod:scsi_io_completion+0xf8/0x2d3
    [] blk_done_softirq+0x62/0x6f
    [] __do_softirq+0x59/0xc3
    [] call_softirq+0x1c/0x28
    [] do_softirq+0x2c/0x7d
    [] irq_exit+0x36/0x81
    [] do_IRQ+0x13d/0x161
    [] ret_from_intr+0x0/0xa
    [] rocessor:acpi_processor_idle+0x2cb/0x496
    [] rocessor:acpi_processor_idle+0x2c7/0x496
    [] rocessor:acpi_processor_idle+0x0/0x496
    [] default_idle+0x0/0x3d
    [] cpu_idle+0x90/0xcc
    [] start_kernel+0x2ab/0x2b7
    [] _sinittext+0x11b/0x122


    Code: 49 8b 40 20 4d 8d 50 20 4c 89 c7 fc b9 08 00 00 00 4c 89 c3
    RIP [] blk_rq_map_sg+0xf6/0x16a
    RSP
    CR2: 0000000000000020
    Kernel panic - not syncing: Aiee, killing interrupt handler!

    Luca
    --
    The trouble with computers is that they do what you tell them,
    not what you want.
    D. Cohen
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  15. Re: [bug] ata subsystem related crash with latest -git



    On Wed, 17 Oct 2007, Jens Axboe wrote:
    >
    > So until that is fixed up, how about this? Should cover all block
    > devices, and I don't think sg_next()/for_each_sg() has spread outside of
    > that yet.


    Heh. Not good. There are drivers that set max phys segmsnts to 1. Either
    through some host-specific setting (MMC), or explicitly (eg a grep shows
    viodasd uses just a constant 1 there).

    Linus
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  16. Re: [bug] ata subsystem related crash with latest -git

    On Wed, Oct 17 2007, Linus Torvalds wrote:
    >
    >
    > On Wed, 17 Oct 2007, Jens Axboe wrote:
    > >
    > > So until that is fixed up, how about this? Should cover all block
    > > devices, and I don't think sg_next()/for_each_sg() has spread outside of
    > > that yet.

    >
    > Heh. Not good. There are drivers that set max phys segmsnts to 1. Either
    > through some host-specific setting (MMC), or explicitly (eg a grep shows
    > viodasd uses just a constant 1 there).


    That would hurt... Care to commit your for_each_sg() uglification fixup
    for now then? Or disable the allocation debug config entry, so that the
    sg+1 deref wont crash? Whichever your prefer, just don't want to cause
    crash pains for the unfortunate souls that happen to run into this
    before the real fix is done.

    --
    Jens Axboe

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  17. Re: [bug] ata subsystem related crash with latest -git


    * Jens Axboe wrote:

    > --- a/block/ll_rw_blk.c
    > +++ b/block/ll_rw_blk.c
    > @@ -631,7 +631,7 @@ void blk_queue_max_phys_segments(struct request_queue *q,
    > printk("%s: set to minimum %d\n", __FUNCTION__, max_segments);
    > }
    >
    > - q->max_phys_segments = max_segments;
    > + q->max_phys_segments = max_segments - 1;


    this one crashes too, if added ontop of all the other fixes so far.

    Ingo

    [ 34.711185] EXT3-fs: mounted filesystem with ordered data mode.
    [ 34.716991] VFS: Mounted root (ext3 filesystem) readonly.
    [ 34.722712] Freeing unused kernel memory: 372k freed
    [ 36.112742] BUG: unable to handle kernel paging request at virtual address 7c8b1000
    [ 36.120246] printing eip: 784e9490 *pde = 00dda027 *pte = 048b1000
    [ 36.126486] Oops: 0000 [#1] DEBUG_PAGEALLOC
    [ 36.130645]
    [ 36.132119] Pid: 0, comm: swapper Not tainted (2.6.23 #15)
    [ 36.137579] EIP: 0060:[<784e9490>] EFLAGS: 00010006 CPU: 0
    [ 36.143043] EIP is at ata_qc_issue+0xd0/0x340
    [ 36.147371] EAX: 3f896000 EBX: 7c8b1000 ECX: 00000008 EDX: 00000008
    [ 36.153610] ESI: 7c86de80 EDI: 7c8b0f80 EBP: 7b54007c ESP: 78a13e28
    [ 36.159851] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
    [ 36.165223] Process swapper (pid: 0, ti=78a12000 task=789753e0 task.ti=78a12000)
    [ 36.172416] Stack: 7c8b1000 7b540000 7b540000 00000000 7c8b0ff0 7b54007c 7c86de80 7b5417a4
    [ 36.180735] 784c24a0 784ef62e 784f2183 7b52de98 7c86de80 7b540000 7b5417a4 7c86de80
    [ 36.189055] 7b540000 7b524004 784f2270 784ef310 784c24a0 7c86de80 00000206 7b524004
    [ 36.197374] Call Trace:
    [ 36.199975] [<784c24a0>] scsi_done+0x0/0x20
    [ 36.204221] [<784ef62e>] ata_scsi_translate+0xbe/0x140
    [ 36.209420] [<784f2183>] ata_scsi_queuecmd+0x33/0x200
    [ 36.214533] [<784f2270>] ata_scsi_queuecmd+0x120/0x200
    [ 36.219733] [<784ef310>] ata_scsi_rw_xlat+0x0/0x220
    [ 36.224672] [<784c24a0>] scsi_done+0x0/0x20
    [ 36.228918] [<784c2d22>] scsi_dispatch_cmd+0x152/0x290
    [ 36.234118] [<78135c67>] trace_hardirqs_on+0x67/0xb0
    [ 36.239145] [<784c8c4e>] scsi_request_fn+0x1be/0x370
    [ 36.244171] [<78408096>] blk_run_queue+0x36/0x80
    [ 36.248850] [<784c7530>] scsi_next_command+0x30/0x50
    [ 36.253877] [<784c76bb>] scsi_end_request+0xab/0xe0
    [ 36.258817] [<784c83c9>] scsi_io_completion+0xa9/0x3d0
    [ 36.264016] [<78135c67>] trace_hardirqs_on+0x67/0xb0
    [ 36.269043] [<78405125>] blk_done_softirq+0x45/0x80
    [ 36.273983] [<78405153>] blk_done_softirq+0x73/0x80
    [ 36.278922] [<7811d4c3>] __do_softirq+0x53/0xb0
    [ 36.283515] [<7811d588>] do_softirq+0x68/0x70
    [ 36.287935] [<78105351>] do_IRQ+0x51/0x90
    [ 36.292008] [<78135c9c>] trace_hardirqs_on+0x9c/0xb0
    [ 36.297034] [<7810388e>] common_interrupt+0x2e/0x40
    [ 36.301974] [<78100c55>] cpu_idle+0x35/0x60
    [ 36.306220] [<78a14b35>] start_kernel+0x265/0x300
    [ 36.310987] [<78a14380>] unknown_bootoption+0x0/0x1e0
    [ 36.316100] =======================
    [ 36.319653] Code: 84 d9 01 00 00 7e 32 31 d2 89 f6 8b 1c 24 83 c2 01 8b 03 2b 05 f8 ec d7 78 c1 f8 05 c1 e0 0c 03 43 04 89 43 08 83 c3 10 89 1c 24 <8b> 03 a8 01 0f 85 58 02 00 00 39 ca 75 d2 f0 83 44 24 00 00 85
    [ 36.338458] EIP: [<784e9490>] ata_qc_issue+0xd0/0x340 SS:ESP 0068:78a13e28
    [ 36.345305] Kernel panic - not syncing: Fatal exception in interrupt
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  18. Re: [bug] ata subsystem related crash with latest -git



    On Wed, 17 Oct 2007, Jens Axboe wrote:
    >
    > That would hurt... Care to commit your for_each_sg() uglification fixup
    > for now then? Or disable the allocation debug config entry, so that the
    > sg+1 deref wont crash?


    Well, in practice, it will only crash with DEBUG_PAGEALLOC, so few enough
    are going to be hit by it. In that sense I don't think we're in any deep
    trouble yet.

    That said, maybe this is an acceptable, if hacky, replacement for the
    current "for_each_sg()" loop.

    It does:
    - starts at one *before* the sglist
    - does the sg_next() at the *top* of the loop rather than the bottom of it
    - has a "--count" before that sg_next, so that we don't do it for the
    case when we break out and have used up all segments.

    Totally untested, but it *may* work, and it doesn't look horribly ugly.

    Ingo, does this actually make any difference?

    Linus

    ---
    include/linux/scatterlist.h | 9 ++++++++-
    1 files changed, 8 insertions(+), 1 deletions(-)

    diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h
    index 2dc7464..f5c8e11 100644
    --- a/include/linux/scatterlist.h
    +++ b/include/linux/scatterlist.h
    @@ -51,11 +51,18 @@ static inline struct scatterlist *sg_next(struct scatterlist *sg)
    return sg;
    }

    +static inline struct scatterlist *sg_safe_next(struct scatterlist *sg, int left)
    +{
    + if (left < 0)
    + return NULL;
    + return sg_next(sg);
    +}
    +
    /*
    * Loop over each sg element, following the pointer to a new list if necessary
    */
    #define for_each_sg(sglist, sg, nr, __i) \
    - for (__i = 0, sg = (sglist); __i < (nr); __i++, sg = sg_next(sg))
    + for (__i = (nr), sg = (sglist)-1; (sg = sg_safe_next(sg, --__i)) != NULL ; )

    /**
    * sg_last - return the last scatterlist entry in a list
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  19. Re: [bug] ata subsystem related crash with latest -git

    On Wed, 17 Oct 2007 14:11:34 -0700 (PDT)
    Linus Torvalds wrote:

    >
    >
    > On Wed, 17 Oct 2007, Jens Axboe wrote:
    > >
    > > That would hurt... Care to commit your for_each_sg() uglification fixup
    > > for now then? Or disable the allocation debug config entry, so that the
    > > sg+1 deref wont crash?

    >
    > Well, in practice, it will only crash with DEBUG_PAGEALLOC, so few enough
    > are going to be hit by it. In that sense I don't think we're in any deep
    > trouble yet.
    >
    > That said, maybe this is an acceptable, if hacky, replacement for the
    > current "for_each_sg()" loop.
    >
    > It does:
    > - starts at one *before* the sglist
    > - does the sg_next() at the *top* of the loop rather than the bottom of it
    > - has a "--count" before that sg_next, so that we don't do it for the
    > case when we break out and have used up all segments.
    >
    > Totally untested, but it *may* work, and it doesn't look horribly ugly.
    >
    > Ingo, does this actually make any difference?
    >
    > Linus
    >
    > ---
    > include/linux/scatterlist.h | 9 ++++++++-
    > 1 files changed, 8 insertions(+), 1 deletions(-)
    >
    > diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h
    > index 2dc7464..f5c8e11 100644
    > --- a/include/linux/scatterlist.h
    > +++ b/include/linux/scatterlist.h
    > @@ -51,11 +51,18 @@ static inline struct scatterlist *sg_next(struct scatterlist *sg)
    > return sg;
    > }
    >
    > +static inline struct scatterlist *sg_safe_next(struct scatterlist *sg, int left)
    > +{
    > + if (left < 0)
    > + return NULL;
    > + return sg_next(sg);
    > +}
    > +
    > /*
    > * Loop over each sg element, following the pointer to a new list if necessary
    > */
    > #define for_each_sg(sglist, sg, nr, __i) \
    > - for (__i = 0, sg = (sglist); __i < (nr); __i++, sg = sg_next(sg))
    > + for (__i = (nr), sg = (sglist)-1; (sg = sg_safe_next(sg, --__i)) != NULL ; )


    Looks that (sglist) - 1 isn't initialized and we use sg_next for it?
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  20. Re: [bug] ata subsystem related crash with latest -git



    On Thu, 18 Oct 2007, FUJITA Tomonori wrote:
    >
    > Looks that (sglist) - 1 isn't initialized and we use sg_next for it?


    sg_next() - as it stands now - never actually looks at the SG that its
    argument points to: it explicitly *only* looks at the next one.

    That's the bug. If sg_next() looked at the actual *current* sg entry, we
    wouldn't have any issues to begin with, and that's what I'm arguing we
    should do in the longer run (where "longer run" is defined as "when Jens
    does it asap").

    So the hacky solution as it stands right now is to actually use the
    behaviour of "sg_next()" to our advantage in for_each_sg(), and starting
    off by setting sg to (sglist)-1. That way we can do the "sg_next()" (which
    will *not* look at the uninitialized space before the array) before
    entering the loop, regardless of whether it's the first time through the
    loop or not.

    So no, it's not technically "strictly conforming ANSI C", because we use a
    pointer (not do not dereference it!) outside of its allocation, which is
    strictly speaking against the standard. However, the kernel does those
    kinds of things all the time, since the kernel does its own memory
    allocation, so that's not actually relevant.

    The *standard* may say that you cannot decrement a pointer to before the
    beginning of the object and then increment it back up again and be
    strictly conforming, but the fact is, we very much depend on contiguous
    and flat kernel pointer model, and always have (and probably always will)

    Linus
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread
Page 3 of 8 FirstFirst 1 2 3 4 5 ... LastLast