Delays imposed by console connection - Embedded

This is a discussion on Delays imposed by console connection - Embedded ; I am trying to understand what the impact is when using the console for error reporting, and what might be the best method for obtaining run-time debug info. For example many embedded boards use an RS232 serial console, and the ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: Delays imposed by console connection

  1. Delays imposed by console connection


    I am trying to understand what the impact is when using the console for
    error reporting, and what might be the best method for obtaining
    run-time debug info. For example many embedded boards use an RS232
    serial console, and the default baud on this is usu no more than 19200.
    If the console is used for debug messages, it would seem that there
    would be some negative impact on application performance, and/or lost
    messages particularly if the buffers can't keep up.

    On my uClinux dev platform I can also telnet in to get a console. The
    connection is potentially much faster, but that slow serial console is
    still being serviced whether or not a cable is attached to it.

    Now maybe the drivers controlling the serial console are written in such
    a way that they don't impose a big hit even when overflowing (no
    blocking), and they don't slow down other connections (telnet). But if
    there is any hit, it would seem to be a good idea to explicitly disable
    the serial console and rely only the telnet connection. A further
    improvement might be to have everything go directly to a log file
    instead of to the console.

    Does redirection take care of this?

    Comments?


    BTW, I don't see anything in the rc or inittab files that would allow me
    to shut down the serial console, but its probably there somewhere.





  2. Re: Delays imposed by console connection

    On Mar 17, 5:52 am, tns1 wrote:
    > I am trying to understand what the impact is when using the console for
    > error reporting, and what might be the best method for obtaining
    > run-time debug info. For example many embedded boards use an RS232
    > serial console, and the default baud on this is usu no more than 19200.
    > If the console is used for debug messages, it would seem that there
    > would be some negative impact on application performance, and/or lost
    > messages particularly if the buffers can't keep up.
    >
    > On my uClinux dev platform I can also telnet in to get a console. The
    > connection is potentially much faster, but that slow serial console is
    > still being serviced whether or not a cable is attached to it.
    >
    > Now maybe the drivers controlling the serial console are written in such
    > a way that they don't impose a big hit even when overflowing (no
    > blocking), and they don't slow down other connections (telnet). But if
    > there is any hit, it would seem to be a good idea to explicitly disable
    > the serial console and rely only the telnet connection. A further
    > improvement might be to have everything go directly to a log file
    > instead of to the console.
    >
    > Does redirection take care of this?
    >
    > Comments?
    >
    > BTW, I don't see anything in the rc or inittab files that would allow me
    > to shut down the serial console, but its probably there somewhere.


    I am running an embedded linux board with a debug serial port at 9600b/
    s, and have been noticing slowness even when I logged on from a SSH
    session specially when there are lot of debug message. And the serial
    console seems to drop messages as well. It will be good if anyone can
    enlighten us in regards to the default debug serial console on linux
    and whether it slows down everything.

  3. Re: Delays imposed by console connection

    Janaka wrote:
    > I am running an embedded linux board with a debug serial port at 9600b/
    > s, and have been noticing slowness even when I logged on from a SSH
    > session specially when there are lot of debug message. And the serial
    > console seems to drop messages as well. It will be good if anyone can
    > enlighten us in regards to the default debug serial console on linux
    > and whether it slows down everything.


    On a full linux install, the initial baud rate may first be set by the
    bios (if there is one), next the bootloader (if you have one) may set it
    differently, then when the kernel boots its own default baud will take
    effect, and finally the application may also set it. I believe the
    bootloader can override the kernel default baud on some boards. For
    instance you may be able to edit the grub.conf file (or lilo, or
    redboot, or uboot) to get a higher baud without touching the kernel.
    9600 seems slow for any processor capable of running linux.

    If your system has no obvious bios or bootloader, you need to dig.

  4. Re: Delays imposed by console connection

    Janaka wrote:

    > On Mar 17, 5:52 am, tns1 wrote:
    >> I am trying to understand what the impact is when using the console for
    >> error reporting, and what might be the best method for obtaining
    >> run-time debug info. For example many embedded boards use an RS232
    >> serial console, and the default baud on this is usu no more than 19200.
    >> If the console is used for debug messages, it would seem that there
    >> would be some negative impact on application performance, and/or lost
    >> messages particularly if the buffers can't keep up.
    >>
    >> On my uClinux dev platform I can also telnet in to get a console. The
    >> connection is potentially much faster, but that slow serial console is
    >> still being serviced whether or not a cable is attached to it.
    >>
    >> Now maybe the drivers controlling the serial console are written in such
    >> a way that they don't impose a big hit even when overflowing (no
    >> blocking), and they don't slow down other connections (telnet). But if
    >> there is any hit, it would seem to be a good idea to explicitly disable
    >> the serial console and rely only the telnet connection. A further
    >> improvement might be to have everything go directly to a log file
    >> instead of to the console.
    >>
    >> Does redirection take care of this?
    >>
    >> Comments?
    >>
    >> BTW, I don't see anything in the rc or inittab files that would allow me
    >> to shut down the serial console, but its probably there somewhere.

    >
    > I am running an embedded linux board with a debug serial port at 9600b/
    > s, and have been noticing slowness even when I logged on from a SSH
    > session specially when there are lot of debug mes
    > sage.


    What kind of debug messages? Kernel or userland?

    > And the serial
    > console seems to drop messages as well. It will be good if anyone can
    > enlighten us in regards to the default debug serial console on linux
    > and whether it slows down everything.


    You define the system console the kernel should output its messages with
    the "console=ttyS0," kernel parameter for a serial
    console. Or anything different device if your target supports it.

    "$ man setconsole" could also be helpful.

    JB


  5. Re: Delays imposed by console connection

    On Mar 17, 7:54 pm, Juergen Beisert wrote:
    > Janaka wrote:
    > > On Mar 17, 5:52 am, tns1 wrote:
    > >> I am trying to understand what the impact is when using the console for
    > >> error reporting, and what might be the best method for obtaining
    > >> run-time debug info. For example many embedded boards use an RS232
    > >> serial console, and the default baud on this is usu no more than 19200.
    > >> If the console is used for debug messages, it would seem that there
    > >> would be some negative impact on application performance, and/or lost
    > >> messages particularly if the buffers can't keep up.

    >
    > >> On my uClinux dev platform I can also telnet in to get a console. The
    > >> connection is potentially much faster, but that slow serial console is
    > >> still being serviced whether or not a cable is attached to it.

    >
    > >> Now maybe the drivers controlling the serial console are written in such
    > >> a way that they don't impose a big hit even when overflowing (no
    > >> blocking), and they don't slow down other connections (telnet). But if
    > >> there is any hit, it would seem to be a good idea to explicitly disable
    > >> the serial console and rely only the telnet connection. A further
    > >> improvement might be to have everything go directly to a log file
    > >> instead of to the console.

    >
    > >> Does redirection take care of this?

    >
    > >> Comments?

    >
    > >> BTW, I don't see anything in the rc or inittab files that would allow me
    > >> to shut down the serial console, but its probably there somewhere.

    >
    > > I am running an embedded linux board with a debug serial port at 9600b/
    > > s, and have been noticing slowness even when I logged on from a SSH
    > > session specially when there are lot of debug mes
    > > sage.

    >
    > What kind of debug messages? Kernel or userland?
    >
    > > And the serial
    > > console seems to drop messages as well. It will be good if anyone can
    > > enlighten us in regards to the default debug serial console on linux
    > > and whether it slows down everything.

    >
    > You define the system console the kernel should output its messages with
    > the "console=ttyS0," kernel parameter for a serial
    > console. Or anything different device if your target supports it.
    >
    > "$ man setconsole" could also be helpful.
    >
    > JB


    Thanks tns1 and JB.
    I use Uboot, and as JB mentioned I set the board rate using
    "console=....". Debug messages are mainly from kernel space where I'm
    building a device driver.
    I can change the board rate for the system console to something
    higher, but still want to know whether this could slow things down ?
    I am SSHing in and then running a user space app to excercise the
    kernel module I am developing (via IOCTL). At this stage of
    development I've put heaps of debug (printk) on the kernel module.
    All (almost all) of these ends up on the system console. I can see
    the dropped debug messages using the dmesg.

  6. Re: Delays imposed by console connection

    Janaka wrote:

    > On Mar 17, 7:54 pm, Juergen Beisert wrote:
    >> Janaka wrote:
    >> > On Mar 17, 5:52 am, tns1 wrote:
    >> >> I am trying to understand what the impact is when using the console
    >> >> for error reporting, and what might be the best method for obtaining
    >> >> run-time debug info. For example many embedded boards use an RS232
    >> >> serial console, and the default baud on this is usu no more than
    >> >> 19200. If the console is used for debug messages, it would seem that
    >> >> there would be some negative impact on application performance, and/or
    >> >> lost messages particularly if the buffers can't keep up.

    >>
    >> >> On my uClinux dev platform I can also telnet in to get a console. The
    >> >> connection is potentially much faster, but that slow serial console is
    >> >> still being serviced whether or not a cable is attached to it.

    >>
    >> >> Now maybe the drivers controlling the serial console are written in
    >> >> such a way that they don't impose a big hit even when overflowing (no
    >> >> blocking), and they don't slow down other connections (telnet). But if
    >> >> there is any hit, it would seem to be a good idea to explicitly
    >> >> disable the serial console and rely only the telnet connection. A
    >> >> further improvement might be to have everything go directly to a log
    >> >> file instead of to the console.

    >>
    >> >> Does redirection take care of this?

    >>
    >> >> Comments?

    >>
    >> >> BTW, I don't see anything in the rc or inittab files that would allow
    >> >> me to shut down the serial console, but its probably there somewhere.

    >>
    >> > I am running an embedded linux board with a debug serial port at 9600b/
    >> > s, and have been noticing slowness even when I logged on from a SSH
    >> > session specially when there are lot of debug mes
    >> > sage.

    >>
    >> What kind of debug messages? Kernel or userland?
    >>
    >> > And the serial
    >> > console seems to drop messages as well. It will be good if anyone can
    >> > enlighten us in regards to the default debug serial console on linux
    >> > and whether it slows down everything.

    >>
    >> You define the system console the kernel should output its messages with
    >> the "console=ttyS0," kernel parameter for a
    >> serial console. Or anything different device if your target supports it.
    >>
    >> "$ man setconsole" could also be helpful.
    >>
    >> JB

    >
    > Thanks tns1 and JB.
    > I use Uboot, and as JB mentioned I set the board rate using
    > "console=....". Debug messages are mainly from kernel space where I'm
    > building a device driver.
    > I can change the board rate for the system console to something
    > higher, but still want to know whether this could slow things down ?
    > I am SSHing in and then running a user space app to excercise the
    > kernel module I am developing (via IOCTL). At this stage of
    > development I've put heaps of debug (printk) on the kernel module.


    Yes, this will drop down the speed. Try this:

    Use
    "printk(KERN_DEBUG"");"
    instead of a plain
    "printk("");"

    Then you can check the impact of your debug messages by running "dmesg -n 1"
    and "dmesg -n 8". (Or you can use all the other KERN_* macros in kernel.h
    to control the noise your driver emits.)

    JB


  7. Re: Delays imposed by console connection

    On Mar 18, 6:55 pm, Juergen Beisert wrote:
    > Janaka wrote:
    > > On Mar 17, 7:54 pm, Juergen Beisert wrote:
    > >> Janaka wrote:
    > >> > On Mar 17, 5:52 am, tns1 wrote:
    > >> >> I am trying to understand what the impact is when using theconsole
    > >> >> for error reporting, and what might be the best method for obtaining
    > >> >> run-time debug info. For example many embedded boards use an RS232
    > >> >> serialconsole, and thedefaultbaud on this is usu no more than
    > >> >> 19200. If theconsoleis used for debug messages, it would seem that
    > >> >> there would be some negative impact on application performance, and/or
    > >> >> lost messages particularly if the buffers can't keep up.

    >
    > >> >> On my uClinux dev platform I can also telnet in to get aconsole. The
    > >> >> connection is potentially much faster, but that slow serialconsoleis
    > >> >> still being serviced whether or not a cable is attached to it.

    >
    > >> >> Now maybe the drivers controlling the serialconsoleare written in
    > >> >> such a way that they don't impose a big hit even when overflowing (no
    > >> >> blocking), and they don't slow down other connections (telnet). But if
    > >> >> there is any hit, it would seem to be a good idea to explicitly
    > >> >> disable the serialconsoleand rely only the telnet connection. A
    > >> >> further improvement might be to have everything go directly to a log
    > >> >> file instead of to theconsole.

    >
    > >> >> Does redirection take care of this?

    >
    > >> >> Comments?

    >
    > >> >> BTW, I don't see anything in the rc or inittab files that would allow
    > >> >> me to shut down the serialconsole, but its probably there somewhere.

    >
    > >> > I am running an embedded linux board with a debug serial port at 9600b/
    > >> > s, and have been noticing slowness even when I logged on from a SSH
    > >> > session specially when there are lot of debug mes
    > >> > sage.

    >
    > >> What kind of debug messages? Kernel or userland?

    >
    > >> > And the serial
    > >> >consoleseems to drop messages as well. It will be good if anyone can
    > >> > enlighten us in regards to thedefaultdebug serialconsoleon linux
    > >> > and whether it slows down everything.

    >
    > >> You define the systemconsolethe kernel should output its messages with
    > >> the "console=ttyS0," kernel parameter for a
    > >> serialconsole. Or anything different device if your target supports it.

    >
    > >> "$ man setconsole" could also be helpful.

    >
    > >> JB

    >
    > > Thanks tns1 and JB.
    > > I use Uboot, and as JB mentioned I set the board rate using
    > > "console=....". Debug messages are mainly from kernel space where I'm
    > > building a device driver.
    > > I can change the board rate for the systemconsoleto something
    > > higher, but still want to know whether this could slow things down ?
    > > I am SSHing in and then running a user space app to excercise the
    > > kernel module I am developing (via IOCTL). At this stage of
    > > development I've put heaps of debug (printk) on the kernel module.

    >
    > Yes, this will drop down the speed. Try this:
    >
    > Use
    > "printk(KERN_DEBUG"");"
    > instead of a plain
    > "printk("");"
    >
    > Then you can check the impact of your debug messages by running "dmesg -n 1"
    > and "dmesg -n 8". (Or you can use all the other KERN_* macros in kernel.h
    > to control the noise your driver emits.)
    >
    > JB


    I was getting odd pauses( up to 100mS) on my data streaming device
    driver and found out that the cause was the slow default console. My
    device driver does about 24-80mb/s using DMA, but grounded to a halt
    while debug messages were put out to the default console at 9600b/s.
    What was more interesting was that default console writes disabled
    Interrupts until the debug messages were put out; which was a killer
    in my data streaming app as it stopped streaming for up to 100mS. So
    if you are doing embedded Linux development be AWARE of this!
    One way to overcome this was to disable serial console output. I will
    be glad to hear any any other suggestions to overcome this.
    BTW I thought about increasing the serial data rate to 115K but that
    would only reduce the ISR lockout time to ~10mS, which is a awfully
    long time in embedded systems sense.

  8. Re: Delays imposed by console connection

    On May 14, 11:04 am, Janaka wrote:
    > On Mar 18, 6:55 pm, Juergen Beisert wrote:
    >
    >
    >
    > > Janaka wrote:
    > > > On Mar 17, 7:54 pm, Juergen Beisert wrote:
    > > >> Janaka wrote:
    > > >> > On Mar 17, 5:52 am, tns1 wrote:
    > > >> >> I am trying to understand what the impact is when using theconsole
    > > >> >> for error reporting, and what might be the best method for obtaining
    > > >> >> run-time debug info. For example many embedded boards use an RS232
    > > >> >> serialconsole, and thedefaultbaud on this is usu no more than
    > > >> >> 19200. If theconsoleis used for debug messages, it would seem that
    > > >> >> there would be some negative impact on application performance, and/or
    > > >> >> lost messages particularly if the buffers can't keep up.

    >
    > > >> >> On my uClinux dev platform I can also telnet in to get aconsole. The
    > > >> >> connection is potentially much faster, but that slow serialconsoleis
    > > >> >> still being serviced whether or not a cable is attached to it.

    >
    > > >> >> Now maybe the drivers controlling the serialconsoleare written in
    > > >> >> such a way that they don't impose a big hit even when overflowing (no
    > > >> >> blocking), and they don't slow down other connections (telnet). But if
    > > >> >> there is any hit, it would seem to be a good idea to explicitly
    > > >> >> disable the serialconsoleand rely only the telnet connection. A
    > > >> >> further improvement might be to have everything go directly to a log
    > > >> >> file instead of to theconsole.

    >
    > > >> >> Does redirection take care of this?

    >
    > > >> >> Comments?

    >
    > > >> >> BTW, I don't see anything in the rc or inittab files that would allow
    > > >> >> me to shut down the serialconsole, but its probably there somewhere.

    >
    > > >> > I am running an embedded linux board with a debug serial port at 9600b/
    > > >> > s, and have been noticing slowness even when I logged on from a SSH
    > > >> > session specially when there are lot of debug mes
    > > >> > sage.

    >
    > > >> What kind of debug messages? Kernel or userland?

    >
    > > >> > And the serial
    > > >> >consoleseems to drop messages as well. It will be good if anyone can
    > > >> > enlighten us in regards to thedefaultdebug serialconsoleon linux
    > > >> > and whether it slows down everything.

    >
    > > >> You define the systemconsolethe kernel should output its messages with
    > > >> the "console=ttyS0," kernel parameter for a
    > > >> serialconsole. Or anything different device if your target supports it.

    >
    > > >> "$ man setconsole" could also be helpful.

    >
    > > >> JB

    >
    > > > Thanks tns1 and JB.
    > > > I use Uboot, and as JB mentioned I set the board rate using
    > > > "console=....". Debug messages are mainly from kernel space where I'm
    > > > building a device driver.
    > > > I can change the board rate for the systemconsoleto something
    > > > higher, but still want to know whether this could slow things down ?
    > > > I am SSHing in and then running a user space app to excercise the
    > > > kernel module I am developing (via IOCTL). At this stage of
    > > > development I've put heaps of debug (printk) on the kernel module.

    >
    > > Yes, this will drop down the speed. Try this:

    >
    > > Use
    > > "printk(KERN_DEBUG"");"
    > > instead of a plain
    > > "printk("");"

    >
    > > Then you can check the impact of your debug messages by running "dmesg -n 1"
    > > and "dmesg -n 8". (Or you can use all the other KERN_* macros in kernel.h
    > > to control the noise your driver emits.)

    >
    > > JB

    >
    > I was getting odd pauses( up to 100mS) on my data streaming device
    > driver and found out that the cause was the slow default console. My
    > device driver does about 24-80mb/s using DMA, but grounded to a halt
    > while debug messages were put out to the default console at 9600b/s.
    > What was more interesting was that default console writes disabled
    > Interrupts until the debug messages were put out; which was a killer
    > in my data streaming app as it stopped streaming for up to 100mS. So
    > if you are doing embedded Linux development be AWARE of this!
    > One way to overcome this was to disable serial console output. I will
    > be glad to hear any any other suggestions to overcome this.
    > BTW I thought about increasing the serial data rate to 115K but that
    > would only reduce the ISR lockout time to ~10mS, which is a awfully
    > long time in embedded systems sense.


    And I have also considered using UDP streaming as my default console
    (eg: netconsole=....), but wanted something that starts up a bit
    before network interfaces get initialised.
    I found the following thread most interesting:
    <http://groups.google.com.au/group/co...ed/browse_frm/
    thread/75fd6ff318e608fb/b28f934eb0cb8f7f?hl=en&lnk=gst&q=default
    +console#b28f934eb0cb8f7f>

+ Reply to Thread