Suggestions for custom application-layer protocol? - Embedded

This is a discussion on Suggestions for custom application-layer protocol? - Embedded ; In article , Glyn Davies wrote: :For simplicity of implementation, I'd recommend implementing a message :layer over the TCP socket - something as simple as a two byte binary header :to indicate the length of the message to follow. :This ...

+ Reply to Thread
Page 2 of 3 FirstFirst 1 2 3 LastLast
Results 21 to 40 of 56

Thread: Suggestions for custom application-layer protocol?

  1. Re: Suggestions for custom application-layer protocol?

    In article ,
    Glyn Davies wrote:
    :For simplicity of implementation, I'd recommend implementing a message
    :layer over the TCP socket - something as simple as a two byte binary header
    :to indicate the length of the message to follow.

    :This means you can then read a whole message - text based or otherwise,
    :and the process it accordingly. You don't need to mess about with parsing
    :lines and stuff like that.

    You still need to "mess about" if your data might not fit in a single
    packet. Recall that when you read() from a socket you are not certain
    to get all of the data unless the data length is no greater than
    your system's atomicity limits [PIPE_BUF]. The POSIX minimum
    value for PIPE_BUF is 512 -- so in the general case a write() of
    more than 512 bytes may require multiple read() statements.
    --
    Ceci, ce n'est pas une idée.

  2. Re: Suggestions for custom application-layer protocol?


    "Walter Roberson" wrote in message
    news:d753g2$lim$1@canopus.cc.umanitoba.ca...

    > You still need to "mess about" if your data might not fit in a single
    > packet.


    And even if it does.

    > Recall that when you read() from a socket you are not certain
    > to get all of the data unless the data length is no greater than
    > your system's atomicity limits [PIPE_BUF]. The POSIX minimum
    > value for PIPE_BUF is 512 -- so in the general case a write() of
    > more than 512 bytes may require multiple read() statements.


    These atomicity limits don't apply to TCP connections, which have their
    own segmentation rules.

    DS



  3. Re: Suggestions for custom application-layer protocol?

    On 2005-05-26, Walter Roberson wrote:

    >:For simplicity of implementation, I'd recommend implementing a message
    >:layer over the TCP socket - something as simple as a two byte binary header
    >:to indicate the length of the message to follow.
    >
    >:This means you can then read a whole message - text based or otherwise,
    >:and the process it accordingly. You don't need to mess about with parsing
    >:lines and stuff like that.
    >
    > You still need to "mess about" if your data might not fit in a single
    > packet.


    And even if it does.

    > Recall that when you read() from a socket you are not certain
    > to get all of the data unless the data length is no greater
    > than your system's atomicity limits [PIPE_BUF]. The POSIX
    > minimum value for PIPE_BUF is 512 -- so in the general case a
    > write() of more than 512 bytes may require multiple read()
    > statements.


    The data may have passed through a bridge or router that may
    have broken it up even further. It's a bad idea to make any
    assumptions about any correspondance between write() and read()
    calls on a TCP connection. It's just a byte stream: you
    oughtn't depend on any sort of datagram behavior whatsoever.

    A 500 byte "message" may have been sent using a single call to
    write(), but the receiver had better be prepared to call read()
    anywhere from 1 to 500 times.

    --
    Grant Edwards grante Yow! Two with FLUFFO,
    at hold th' BEETS...side of
    visi.com SOYETTES!

  4. Re: Suggestions for custom application-layer protocol?

    "Mark" wrote in message
    news:lb2le.10958$on1.6364@clgrps13...
    > I need to implement a simple application layer protocol that will be used

    to
    > communicate between an embedded device (single-board computer running

    Linux)
    > and a monitoring terminal (running Windows). There isn't alot of data

    being
    > passed around, mostly status information collected by the embedded device,
    > and control messages from the monitoring terminal. Messages will consist

    of
    > between 1 and 10 fields of data.
    >
    > I want the protocol to be text based rather than binary since the data
    > throughput is low. I also want the protocol to be based on TCP/IP. Does
    > anyone here have any suggestions on the design of a simple protocol? Are
    > there simple, standard ways of formatting text messages to be sent over a
    > TCP socket (such as comma-separated)? I would prefer to avoid the
    > complexities of XML. Since either the embedded device or the monitoring
    > terminal can initiate a message, is it preferable to have TCP servers
    > running on both sides? Or is it better to simply leave a TCP connection
    > open between client and server? Any suggestions on ultra-simple standard
    > protocols that do something similar to this?
    >
    > Any suggestions or pointers to further reading are much appreciated.
    >
    > Thanks,
    > Mark
    >
    >


    Thanks alot for all the responses, there are some excellent ideas here which
    I will investigate further. I looked briefly at the BEEP protocol but it
    seems to support alot more features than I need. I control both ends of the
    protocol, so don't need to overdo it on versioning robustness, etc. To
    clarify a few responders questions ... by "monitoring terminals" I meant
    Windows-based computers with a simple GUI that displays the data and allows
    users to control various aspects of the embedded device from a remote
    location. I expect I will end up with a text-based protocol, with a header
    that indicates the number of bytes in the payload. The payloads will be
    composed of field=value pairs.

    Thanks guys I really appreciate it, this discussion has helped me out a
    great deal!

    Mark



  5. Re: Suggestions for custom application-layer protocol?

    On 2005-05-26, Mark wrote:

    > I expect I will end up with a text-based protocol, with a header
    > that indicates the number of bytes in the payload.


    If you're using a text-based protocol, you don't nead a header
    with a payload count to tell you where the message ends. Use
    an end-of-message character or string. The cr/lf pair is used
    as the end-of-message delimiter by a lot of text-based
    runs-on-top of TCP protocols. That approach works wonderfully
    with line-oriented high-level text-handling libraries and
    utilties -- you can do all sorts of useful stuff using nothing
    but a few utilities like netcat, expect, and so on.

    I've also seen protocols that use STX/ETX to frame messages,
    and that works nicely as well, but isn't as compatible with
    pre-existing utilities.

    > The payloads will be composed of field=value pairs.


    That Sounds like a good plan.

    > Thanks guys I really appreciate it, this discussion has helped
    > me out a great deal!


    --
    Grant Edwards grante Yow! Hey, LOOK!! A pair of
    at SIZE 9 CAPRI PANTS!! They
    visi.com probably belong to SAMMY
    DAVIS, JR.!!

  6. Re: Suggestions for custom application-layer protocol?

    On Wed, 25 May 2005 19:21:02 +0000, Sean Burke wrote:

    > One option to consider is to embed a web interface into your
    > application. This has the advantage that you can use any web
    > browser as the client side of the interface.


    Not a terrible idea, a simple HTTP/1.0 server can be pretty small esp. if
    you don't mind stopping as soon as something works (the very basics can be
    done in < 20 lines of C).

    > There are a variety of very small web servers that are suitable
    > for embedding. One such that I have used successfully is "pserv".
    > From the FreeBSD ports description:


    Yeh, pretty good ... I stopped looking after seeing 2 major
    vulnerabilities on the first google page.
    The code also looked promising ... for more exploits.


    Writing a custom "simple protocol" is likely to be much easier, using
    netstrings is probably more likely to make you do the right thing ... but
    a simple "CMD arg1 arg2" type telnet/SMTP/NNTP like protocol isn't hard to
    get right.

    --
    James Antill -- james@and.org
    http://www.and.org/vstr/httpd


  7. Re: Suggestions for custom application-layer protocol?

    On Fri, 27 May 2005 00:04:16 +0000, Grant Edwards wrote:

    > On 2005-05-26, Mark wrote:
    >
    >> I expect I will end up with a text-based protocol, with a header
    >> that indicates the number of bytes in the payload.

    >
    > If you're using a text-based protocol, you don't nead a header
    > with a payload count to tell you where the message ends. Use
    > an end-of-message character or string. The cr/lf pair is used
    > as the end-of-message delimiter by a lot of text-based
    > runs-on-top of TCP protocols. That approach works wonderfully
    > with line-oriented high-level text-handling libraries and
    > utilties -- you can do all sorts of useful stuff using nothing
    > but a few utilities like netcat, expect, and so on.


    It interacts well with telnet, which is about the best thing that can be
    said for it. There are large problems with how you limit "too long" lines,
    and what happens when you hit a CR or LF on it's own. Esp. if you put
    something like a perl regexp directly on the socket.
    The HTTP response splitting problem is entirely the fault of pretending
    CRLF terminated lines are a good idea.

    --
    James Antill -- james@and.org
    http://www.and.org/vstr/httpd


  8. Re: Suggestions for custom application-layer protocol?

    On Fri, 27 May 2005 01:12:46 -0400, James Antill wrote:

    > On Fri, 27 May 2005 00:04:16 +0000, Grant Edwards wrote:
    >
    >> On 2005-05-26, Mark wrote:
    >>
    >>> I expect I will end up with a text-based protocol, with a header
    >>> that indicates the number of bytes in the payload.

    >>
    >> If you're using a text-based protocol, you don't nead a header
    >> with a payload count to tell you where the message ends. Use
    >> an end-of-message character or string. The cr/lf pair is used
    >> as the end-of-message delimiter by a lot of text-based
    >> runs-on-top of TCP protocols. That approach works wonderfully
    >> with line-oriented high-level text-handling libraries and
    >> utilties -- you can do all sorts of useful stuff using nothing
    >> but a few utilities like netcat, expect, and so on.

    >
    > It interacts well with telnet, which is about the best thing that can be
    > said for it. There are large problems with how you limit "too long" lines,
    > and what happens when you hit a CR or LF on it's own. Esp. if you put
    > something like a perl regexp directly on the socket.
    > The HTTP response splitting problem is entirely the fault of pretending
    > CRLF terminated lines are a good idea.


    This is the main reason I'm leaning towards a byte count to delimit the
    message. But I think I'll end up with some concept of special characters
    in the protocol (= , crlf). Is there a standard way to escape these
    characters if they appear in the body of the message? I suppose I could
    use nested netstrings to encode each individual field in the message.
    Thoughts?



  9. Re: Suggestions for custom application-layer protocol?

    In comp.os.linux.embedded Walter Roberson wrote:
    > In article ,
    > Glyn Davies wrote:
    > :For simplicity of implementation, I'd recommend implementing a message
    > :layer over the TCP socket - something as simple as a two byte binary header
    > :to indicate the length of the message to follow.


    > :This means you can then read a whole message - text based or otherwise,
    > :and the process it accordingly. You don't need to mess about with parsing
    > :lines and stuff like that.


    > You still need to "mess about" if your data might not fit in a single
    > packet. Recall that when you read() from a socket you are not certain
    > to get all of the data unless the data length is no greater than
    > your system's atomicity limits [PIPE_BUF]. The POSIX minimum
    > value for PIPE_BUF is 512 -- so in the general case a write() of
    > more than 512 bytes may require multiple read() statements.


    Yes - but you do know how much you are getting.
    You can handle all this stuff in a comms layer, then concentrate on your
    application logic.

    I've done countless systems using this technique. Works well, and is simple.

    Glyn

    --

    ------------------------------------------------------------------------
    Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
    ------------------------------------------------------------------------

  10. Re: Suggestions for custom application-layer protocol?

    In comp.os.linux.embedded Grant Edwards wrote:
    > On 2005-05-26, Mark wrote:


    >> I expect I will end up with a text-based protocol, with a header
    >> that indicates the number of bytes in the payload.


    > If you're using a text-based protocol, you don't nead a header
    > with a payload count to tell you where the message ends. Use
    > an end-of-message character or string. The cr/lf pair is used
    > as the end-of-message delimiter by a lot of text-based
    > runs-on-top of TCP protocols. That approach works wonderfully
    > with line-oriented high-level text-handling libraries and
    > utilties -- you can do all sorts of useful stuff using nothing
    > but a few utilities like netcat, expect, and so on.


    > I've also seen protocols that use STX/ETX to frame messages,
    > and that works nicely as well, but isn't as compatible with
    > pre-existing utilities.




    Most STX/ETX stuff I have seen was over serial comms.
    Not seen it in use over TCP - except where there was an annex box
    feeding out a bunch of serial lines converted to TCP.

    Glyn

    --

    ------------------------------------------------------------------------
    Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
    ------------------------------------------------------------------------

  11. Re: Suggestions for custom application-layer protocol?

    In comp.os.linux.embedded Mark wrote:
    > On Fri, 27 May 2005 01:12:46 -0400, James Antill wrote:




    > This is the main reason I'm leaning towards a byte count to delimit the
    > message. But I think I'll end up with some concept of special characters
    > in the protocol (= , crlf). Is there a standard way to escape these
    > characters if they appear in the body of the message? I suppose I could
    > use nested netstrings to encode each individual field in the message.
    > Thoughts?


    Standard way of escaping a 'special character' is to put it in twice.

    ie, if % was your special character, to have a '%' in your message you'd
    put %%

    If you do put a byte count in, don't put it in as text. It might make
    testing easier, but I doubt it.

    Being able to telnet in and test things is the only advantage of CR/LF
    protocols. Its not hard to knock up test apps which can send a message
    though once you have done your basic comms library (if using a binary header)

    Glyn



    --

    ------------------------------------------------------------------------
    Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
    ------------------------------------------------------------------------

  12. Re: Suggestions for custom application-layer protocol?

    Glyn Davies writes:

    >> I've also seen protocols that use STX/ETX to frame messages,
    >> and that works nicely as well, but isn't as compatible with
    >> pre-existing utilities.

    >
    > Most STX/ETX stuff I have seen was over serial comms.
    > Not seen it in use over TCP - except where there was an annex box
    > feeding out a bunch of serial lines converted to TCP.


    The protocol used by Micros Fidelio point of sale terminals use
    STX/ETX framing when talking to the property management system. The
    reason for this is probably that these devices historically were
    connected with serial links.

    --
    Måns Rullgård
    mru@inprovide.com

  13. Re: Suggestions for custom application-layer protocol?

    In comp.os.linux.embedded M?ns Rullg?rd wrote:
    > Glyn Davies writes:


    >>> I've also seen protocols that use STX/ETX to frame messages,
    >>> and that works nicely as well, but isn't as compatible with
    >>> pre-existing utilities.

    >>
    >> Most STX/ETX stuff I have seen was over serial comms.
    >> Not seen it in use over TCP - except where there was an annex box
    >> feeding out a bunch of serial lines converted to TCP.


    > The protocol used by Micros Fidelio point of sale terminals use
    > STX/ETX framing when talking to the property management system. The
    > reason for this is probably that these devices historically were
    > connected with serial links.



    There is a multi-drop (I think thats the term, my brain is a bit frazzled today)
    which was used to connect many terminals to a host controller.

    ISO/ASYNC I think it was called. Relied on a host polling each terminal
    in turn, at which point they were allowed to speak.

    Was used extensively with serial driven ATMs - most are TCP/IP these days
    I believe.

    But we digress...

    Glyn
    --

    ------------------------------------------------------------------------
    Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
    ------------------------------------------------------------------------

  14. Re: Suggestions for custom application-layer protocol?

    Glyn Davies writes:

    > In comp.os.linux.embedded M?ns Rullg?rd wrote:
    >> Glyn Davies writes:

    >
    >>>> I've also seen protocols that use STX/ETX to frame messages,
    >>>> and that works nicely as well, but isn't as compatible with
    >>>> pre-existing utilities.
    >>>
    >>> Most STX/ETX stuff I have seen was over serial comms.
    >>> Not seen it in use over TCP - except where there was an annex box
    >>> feeding out a bunch of serial lines converted to TCP.

    >
    >> The protocol used by Micros Fidelio point of sale terminals use
    >> STX/ETX framing when talking to the property management system. The
    >> reason for this is probably that these devices historically were
    >> connected with serial links.

    >
    > There is a multi-drop (I think thats the term, my brain is a bit
    > frazzled today) which was used to connect many terminals to a host
    > controller.
    >
    > ISO/ASYNC I think it was called. Relied on a host polling each terminal
    > in turn, at which point they were allowed to speak.


    I've never had the misfortune of having to deal with such a thing.

    > Was used extensively with serial driven ATMs - most are TCP/IP these
    > days I believe.


    I've once seen an ATM display an mswindows dialog box with a DHCP
    related error message.

    --
    Måns Rullgård
    mru@inprovide.com

  15. Re: Suggestions for custom application-layer protocol?



    >> There is a multi-drop (I think thats the term, my brain is a bit
    >> frazzled today) which was used to connect many terminals to a host
    >> controller.
    >>
    >> ISO/ASYNC I think it was called. Relied on a host polling each terminal
    >> in turn, at which point they were allowed to speak.


    > I've never had the misfortune of having to deal with such a thing.


    :-)
    By the time I got to it we were moving people off them to dedicated lines.
    Protocol stayed though as it worked fine.

    >> Was used extensively with serial driven ATMs - most are TCP/IP these
    >> days I believe.


    > I've once seen an ATM display an mswindows dialog box with a DHCP
    > related error message.


    Yeah - they obviously didn't have the clever* stuff we knocked up
    to stop that kind of thing happening.

    Glyn
    --

    ------------------------------------------------------------------------
    Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
    ------------------------------------------------------------------------

  16. Re: Suggestions for custom application-layer protocol?

    On 2005-05-27, James Antill wrote:

    >> If you're using a text-based protocol, you don't nead a header
    >> with a payload count to tell you where the message ends. Use
    >> an end-of-message character or string. The cr/lf pair is used
    >> as the end-of-message delimiter by a lot of text-based
    >> runs-on-top of TCP protocols. That approach works wonderfully
    >> with line-oriented high-level text-handling libraries and
    >> utilties -- you can do all sorts of useful stuff using nothing
    >> but a few utilities like netcat, expect, and so on.

    >
    > It interacts well with telnet,


    If you handle IAC seqeunces -- at least well enough to ignore
    them.

    > which is about the best thing that can be said for it.


    And it's a pretty big thing, in my experience.

    > There are large problems with how you limit "too long" lines,


    What problems? If a message has to be X bytes long, it's X
    bytes long regardless of how you choose to delimit it.
    Handling long messages is a general issue and not specific to
    the delimiter charater chosen.

    > and what happens when you hit a CR or LF on it's own.


    Yes, that can be a problem if you have to be able to include
    arbitrary strings in the message body. I was under the
    impression that this wasn't the case for the OP's application.
    I could be wrong.

    > Esp. if you put something like a perl regexp directly on the
    > socket. The HTTP response splitting problem is entirely the
    > fault of pretending CRLF terminated lines are a good idea.


    My other suggestion to use STX/ETX is pretty simple and is
    quite robust if you impliment an escape mechanism for when
    STX,ETX, or the escape character occur in the message body.

    --
    Grant Edwards grante Yow! Sometime in 1993
    at NANCY SINATRA will lead a
    visi.com BLOODLESS COUP on GUAM!!

  17. Re: Suggestions for custom application-layer protocol?

    On 2005-05-27, Mark wrote:

    >> It interacts well with telnet, which is about the best thing that can be
    >> said for it. There are large problems with how you limit "too long" lines,
    >> and what happens when you hit a CR or LF on it's own. Esp. if you put
    >> something like a perl regexp directly on the socket.
    >> The HTTP response splitting problem is entirely the fault of pretending
    >> CRLF terminated lines are a good idea.

    >
    > This is the main reason I'm leaning towards a byte count to delimit the
    > message. But I think I'll end up with some concept of special characters
    > in the protocol (= , crlf). Is there a standard way to escape these
    > characters if they appear in the body of the message?


    Assuming you're using a 7-bit character set, one of the easiest
    schemes I've seen is on transmit to convert to

    [Escape] |0x80

    Remember to include the [Escape] character in your set of
    special characters that get escaped:

    When sending a message body,

    STX becomes ESC,STX|0x80
    ETX becomes ESC,ETX|0x80
    ESC becomes ESC,ESC|0x80

    When receiving a message body

    ESC, becomes &0x7f

    --
    Grant Edwards grante Yow! Yow! Are you the
    at self-frying president?
    visi.com

  18. Re: Suggestions for custom application-layer protocol?

    On 2005-05-27, Glyn Davies wrote:

    >> I've also seen protocols that use STX/ETX to frame messages,
    >> and that works nicely as well, but isn't as compatible with
    >> pre-existing utilities.

    >
    >
    >
    > Most STX/ETX stuff I have seen was over serial comms.
    > Not seen it in use over TCP - except where there was an annex box
    > feeding out a bunch of serial lines converted to TCP.


    A byte stream is a byte stream. The serial (as in RS-232) byte
    stream isn't reliable, but I cann't see any difference between a
    serial comm link and a TCP link when it comes to message framing.

    --
    Grant Edwards grante Yow! How do I get HOME?
    at
    visi.com

  19. Re: Suggestions for custom application-layer protocol?

    In comp.os.linux.embedded Grant Edwards wrote:
    > On 2005-05-27, Glyn Davies wrote:


    >>> I've also seen protocols that use STX/ETX to frame messages,
    >>> and that works nicely as well, but isn't as compatible with
    >>> pre-existing utilities.

    >>
    >>
    >>
    >> Most STX/ETX stuff I have seen was over serial comms.
    >> Not seen it in use over TCP - except where there was an annex box
    >> feeding out a bunch of serial lines converted to TCP.


    > A byte stream is a byte stream. The serial (as in RS-232) byte
    > stream isn't reliable, but I cann't see any difference between a
    > serial comm link and a TCP link when it comes to message framing.



    Nope, there is no general difference...
    The serial streams were generally 7bit though, which would stop you
    using a binary length header.

    Also, as you point out the serial stream is unreliable.
    That makes a difference in the protocol you choose. With STX/ETX framing if you get
    garbage you can resync at the next STX. If you are using length headers
    and you get some noise on the line, you are lost and have no way to
    resync.

    Glyn

    --

    ------------------------------------------------------------------------
    Glyn Davies / gryn@plig.net / www.technobobbins.com / Insert quote here?
    ------------------------------------------------------------------------

  20. Re: Suggestions for custom application-layer protocol?

    On Fri, 27 May 2005 11:06:12 +0000, Glyn Davies wrote:

    > In comp.os.linux.embedded Mark wrote:
    >> On Fri, 27 May 2005 01:12:46 -0400, James Antill wrote:

    >
    >
    >
    >> This is the main reason I'm leaning towards a byte count to delimit the
    >> message. But I think I'll end up with some concept of special characters
    >> in the protocol (= , crlf). Is there a standard way to escape these
    >> characters if they appear in the body of the message? I suppose I could
    >> use nested netstrings to encode each individual field in the message.
    >> Thoughts?


    Generally I'd say one of the major advantages of having leading byte
    counts is that you don't have to escape anything. Where you need multiple
    arguments just enclose them all in a single netstring. Ie. for abcd = xyz
    do[1]

    13:4:abcd,3:xyz,,

    ....if you want to get really fancy (and allow easier exensibility) you can
    do key/value pairs[2].

    > Standard way of escaping a 'special character' is to put it in twice.
    >
    > ie, if % was your special character, to have a '%' in your message you'd
    > put %%


    That works "ok" if you will only ever have a single character that needs
    escaping, like IAC escaping in telnet ... it works much worse if you have
    to do more than one (CSV is the obvious example here). Then doing
    html/C/url numbered escaping works better (but again, if you can avoid it
    I'd do so).

    > If you do put a byte count in, don't put it in as text. It might make
    > testing easier, but I doubt it.


    Putting a byte count in as a binary encoding should always be a last
    resort. You save little, and you are then tied to that byte size[3].


    [1] http://www.and.org/vstr/ex_netstr.c
    [2] http://www.and.org/vstr/ex_dir_list.c
    [3] http://www.and.org/texts/ascii_binary_data

    --
    James Antill -- james@and.org
    http://www.and.org/vstr/httpd


+ Reply to Thread
Page 2 of 3 FirstFirst 1 2 3 LastLast