[PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console - Kernel

This is a discussion on [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console - Kernel ; This series may help debugging certain circumstances where the serial console is unreponsive (e.g. RT51+ spinner, or scheduler problem). It changes the serial8250 driver to use IRQF_NODELAY so that interrupts execute in irq context instead of a kthread. It works ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console

  1. [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console

    This series may help debugging certain circumstances where the serial
    console is unreponsive (e.g. RT51+ spinner, or scheduler problem). It changes
    the serial8250 driver to use IRQF_NODELAY so that interrupts execute in irq
    context instead of a kthread.

    It works pretty well on this end, though it is admitted not fully baked. For
    instance, a few paths in sysrq can still call into non-raw spinlocks
    (sched_debug_show, for instance) which will cause subsequent errors. Also,
    if you use KDB, be sure to convert the kdb_printf_lock to raw as well. I may
    send a KDB related patch seperately.

    I am sending this out now in case it is helpful to someone.

    Regards,
    -Greg
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  2. Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console

    On Fri, 5 Oct 2007, Gregory Haskins wrote:
    > This series may help debugging certain circumstances where the serial
    > console is unreponsive (e.g. RT51+ spinner, or scheduler problem). It changes
    > the serial8250 driver to use IRQF_NODELAY so that interrupts execute in irq
    > context instead of a kthread.
    >
    > It works pretty well on this end, though it is admitted not fully baked. For
    > instance, a few paths in sysrq can still call into non-raw spinlocks
    > (sched_debug_show, for instance) which will cause subsequent errors. Also,
    > if you use KDB, be sure to convert the kdb_printf_lock to raw as well. I may
    > send a KDB related patch seperately.
    >
    > I am sending this out now in case it is helpful to someone.


    The simple non-patch solution is to up the priority of the serial
    console irq to maximum. That's usually sufficient to catch runnaway tasks
    etc. It does not interfere with the system in normal operation as long as
    you do not hit keys in your minicom.

    I doubt that your patch has a chance to survive lockdep and anything which
    is not a sysrq.

    tglx
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  3. Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console

    On Fri, 2007-10-05 at 18:41 +0200, Thomas Gleixner wrote:
    > On Fri, 5 Oct 2007, Gregory Haskins wrote:
    > > This series may help debugging certain circumstances where the serial
    > > console is unreponsive (e.g. RT51+ spinner, or scheduler problem). It changes
    > > the serial8250 driver to use IRQF_NODELAY so that interrupts execute inirq
    > > context instead of a kthread.
    > >
    > > It works pretty well on this end, though it is admitted not fully bakedFor
    > > instance, a few paths in sysrq can still call into non-raw spinlocks
    > > (sched_debug_show, for instance) which will cause subsequent errors. Also,
    > > if you use KDB, be sure to convert the kdb_printf_lock to raw as well. I may
    > > send a KDB related patch seperately.
    > >
    > > I am sending this out now in case it is helpful to someone.

    >
    > The simple non-patch solution is to up the priority of the serial
    > console irq to maximum. That's usually sufficient to catch runnaway tasks
    > etc. It does not interfere with the system in normal operation as long as
    > you do not hit keys in your minicom.
    >
    > I doubt that your patch has a chance to survive lockdep and anything which
    > is not a sysrq.


    Your point is valid, and I thought of this too. The one problem with it
    is that it is not immune to problems in the scheduler itself (which at
    the time, I thought I was chasing..turned out to be lockdep), but that
    is probably a rarity. In any case, we have been using it on a number of
    debug systems here for a few weeks and the console continues to work
    well in all functions besides sysrq, believe it or not (*). But I don't
    see this as being anything used for real. Just thought I would throw it
    out there in case it was useful or inspired someone to give it some
    love.

    Regards,
    -Greg

    (*) The trick is that once it figures out its not for sysrq, it
    schedules a (kthread based) tasklet to do the actual tty() callbacks.

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.2 (GNU/Linux)

    iD8DBQBHBmw1SW61Ku6jJPkRAncsAKDKxZ2bsR3HcaggQigNbj uvZq+uawCgs0NI
    uBfeYYlT6mXMbukmyoQVNDg=
    =/s93
    -----END PGP SIGNATURE-----


  4. Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console


    --
    On Fri, 5 Oct 2007, Thomas Gleixner wrote:

    > On Fri, 5 Oct 2007, Gregory Haskins wrote:
    > > This series may help debugging certain circumstances where the serial
    > > console is unreponsive (e.g. RT51+ spinner, or scheduler problem). It changes
    > > the serial8250 driver to use IRQF_NODELAY so that interrupts execute in irq
    > > context instead of a kthread.
    > >
    > > It works pretty well on this end, though it is admitted not fully baked. For
    > > instance, a few paths in sysrq can still call into non-raw spinlocks
    > > (sched_debug_show, for instance) which will cause subsequent errors. Also,
    > > if you use KDB, be sure to convert the kdb_printf_lock to raw as well. I may
    > > send a KDB related patch seperately.
    > >
    > > I am sending this out now in case it is helpful to someone.

    >
    > The simple non-patch solution is to up the priority of the serial
    > console irq to maximum. That's usually sufficient to catch runnaway tasks
    > etc. It does not interfere with the system in normal operation as long as
    > you do not hit keys in your minicom.
    >
    > I doubt that your patch has a chance to survive lockdep and anything which
    > is not a sysrq.


    This issue has hit me enough times where I've played with a few other
    ideas. I just haven't had the time to finish them. The main problem is if
    the system locks up somewhere we have a lock held that keeps us from
    scheduling. Once that happens in -rt, we are as good as dead. Since you
    can't get much info out without a JTAG or some other mechanism.

    One thought I had was to create a small handler that could read the serial
    and place data into a buffer. This handler would be a IRQ_NODELAY, and it
    would wake up a serial thread to handle the data just like it would for
    normal serial output. But the serial thread would just read from the
    buffer instead of the serial itself. In the case of a sysrq key coming
    in, the IRQ_NODELAY handler would handle it.

    Just a thought. Maybe someday I can get time to play more with this idea,
    or someone else can come up with one. But to convert the entire handler
    into nodelay is a bit extreme.

    -- Steve

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  5. Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console

    On Mon, 2007-10-08 at 10:10 -0400, Steven Rostedt wrote:

    > This issue has hit me enough times where I've played with a few other
    > ideas. I just haven't had the time to finish them. The main problem is if
    > the system locks up somewhere we have a lock held that keeps us from
    > scheduling. Once that happens in -rt, we are as good as dead. Since you
    > can't get much info out without a JTAG or some other mechanism.
    >
    > One thought I had was to create a small handler that could read the serial
    > and place data into a buffer. This handler would be a IRQ_NODELAY, and it
    > would wake up a serial thread to handle the data just like it would for
    > normal serial output. But the serial thread would just read from the
    > buffer instead of the serial itself. In the case of a sysrq key coming
    > in, the IRQ_NODELAY handler would handle it.
    >


    Hi Steve,
    What you describe is exactly what I did. The IRQF_NODELAY handler
    just minimally checks to see if the character is a sysrq related one (or
    KDB, if you have the KDB patches applied). If it is not, it puts the
    character into a ring-buffer and wakes a (kthread-based) tasklet to do
    the normal serial/tty rx path by processing the ring.

    Hope that helps to clarify.

    Regards,
    -Greg



    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  6. Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console


    --
    On Mon, 8 Oct 2007, Gregory Haskins wrote:
    >
    > Hi Steve,
    > What you describe is exactly what I did. The IRQF_NODELAY handler
    > just minimally checks to see if the character is a sysrq related one (or
    > KDB, if you have the KDB patches applied). If it is not, it puts the
    > character into a ring-buffer and wakes a (kthread-based) tasklet to do
    > the normal serial/tty rx path by processing the ring.


    Ah, that was in the second patch (I didn't look at that. My fault). I
    didn't see this description in the comments. But you are right, it is very
    close to what I did too. But before we add this to -rt, it will need to
    probably need to be tested quite a bit more. Especially since this is more
    to help us developers than to help end users.

    I would like to see something like this eventually make it into RT. Just
    because there has been too many times that this would be useful. For now I
    just use the NMI watchdog and soft lockup detect. But those are very
    limiting.

    >
    > Hope that helps to clarify.


    Yes, thanks!

    -- Steve

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  7. Re: [PATCH 0/2] [RFC] RT: Optionally allow IRQF_NODELAY on serial console

    On Mon, 2007-10-08 at 10:41 -0400, Steven Rostedt wrote:
    > --
    > On Mon, 8 Oct 2007, Gregory Haskins wrote:
    > >
    > > Hi Steve,
    > > What you describe is exactly what I did. The IRQF_NODELAY handler
    > > just minimally checks to see if the character is a sysrq related one (or
    > > KDB, if you have the KDB patches applied). If it is not, it puts the
    > > character into a ring-buffer and wakes a (kthread-based) tasklet to do
    > > the normal serial/tty rx path by processing the ring.

    >
    > Ah, that was in the second patch (I didn't look at that. My fault). I
    > didn't see this description in the comments.


    Well, the lack of a proper description is my fault. I can fix this if
    there is interest in actually including it.

    > But you are right, it is very
    > close to what I did too. But before we add this to -rt, it will need to
    > probably need to be tested quite a bit more.


    Totally agree. Note that it already is configurably inert, and defaults
    to disabled so that might make it work for the general user just fine.
    FWIW, our "debug" kernels have been running with the feature on for
    quite a few weeks without problems. But I would fully expect someone
    external to want to quantify that claim on their own before it would be
    considered. So give it a whirl if you get a chance.

    Regards,
    -Greg

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

+ Reply to Thread