How to talk to your microcontroller over serial

Low information density

Generally, you won't be able to stuff much more than four or five bit of information down a serial port using a single byte of a human-readable protocol. In many cases you will get much less. If you have 10 commands that are only issued a couple times a second nobody cares that you spend maybe ten bytes per command on nice, verbose strings such as SET LED. But if you're trying to squeeze a half-kilobyte framebuffer down the line you might start to notice the difference between hex and base-64 encoding, and a binary protocol might really be more up to the job.

Complex parsing code

On the computer side of thing, with the whole phalanx of an operating system, the standard library of your programming language of choice and for all intents and purposes unlimted CPU and memory resources to spare you can easily parse anything spoken on a serial port in real time, even at a blazing fast full Megabaud. The microcontroller side however is an entirely different beast. On a small microcontroller, printf alone will eat about half your flash. On most small microcontrollers, you just won't get a regex library even though it would make parsing textual commands so much simpler. Lacking these resources, you might end up hand-knitting a lot of low-level C code to do something seemingly simple such as parsing set_channel (13, 1.1333)\n. These issues have to be taken into account in the protocol design from the beginning. For example, you don't really need matching parentheses, don't use them.

Fragile protocol state

Say you have a SET_DISPLAY command. Now say your display can display four lines of text. The obvious approach to this is probably the SMTP or HTTP way of sending SET_DISPLAY\nThis is line 1\nThis is line 2\n\n. This would certainly work, but it is very fragile. With this protocol, you're in trouble if at any point the terminating second newline character gets mangled (say, someone unplugs the cable, or the control computer reboots, or a cosmic ray hits something and 0x10 '\n' turns into 0x50 'P').

Timeouts don't work

You might try to solve the problem of your protocol state machine tangling up with a timeout. "If I don't get a valid command for more than 200ms I go back to default state." But consider the above example. Say, your control computer sends a SET_DISPLAY command every 100ms. If in one of them the state machine tangles up, the parser hangs since the timeout is never hit, because a new line of text is arriving every 100ms.

Framing is hard

You might also try to drop the second newline and using a convention such as SET_DISPLAY is followed by two lines of text, then commands resume.". This works as long as your display contents never look like commands. If you are only ever displaying the same three messages on a character LCD that might work, but if you're displaying binary framebuffer data you've lost.

Keep the state machine simple

In a text-based protocol, always use a single line of text to represent a single command. Don't do protocol states or modes where you can toggle between different interpretations for a line. If you have to send human-readable text as part of a command (such as SET_DISPLAY), escape it so it doesn't contain any newlines.

This way, you keep your protocol state machine simple. If at any time your serial trips and flips a bit or looses a byte your protocol will recover on the next newline character, returning to its base state.

Encode numbers in hex when possible

Printing a number in hexadecimal is a very tidy operation, even on the smalest 8-bit microcontrollers. In contrast, printing decimal requires both division and remainder in a loop which might get annoyingly code- and time-intensive on large numbers (say a 32-bit int) and small microcontrollers.

If you have to send fractional values, consider their precision. Instead of sending a 12 bit ADC result as a 32-bit float formatted like 0.176513671875 sending 0x2d3 and dividing by 4096 on the host might be more sensible. If you really have to communicate big floats and you can't take the overhead of including both printf and scanf you can use hexadecimal floating point, which is basically hex((int)foo) + "." + hex((int)(65536*(foo - (int)foo))) for four digits. You can also just hex-encode the binary IEEE-754 representation of the float, sending hex(*(int *)&float). Most programming languages will have a simple, built-in means to parse this sort of thing.

Escape multiline strings

If you have to send arbitrary strings, escape special characters. This not only has the advantage of yielding a robust protocol: It also ensures you can actually see everything that's going on when debugging. The string "\r\n" is easy to distinguish from "\n" while your terminal emulator might not care.

The simplest encoding to use is the C-style backslash encoding. Host-side, most languages will have a built-in means of escaping a string like that.

SLIP/PPP-like special character framing

Some protocols solve this problem much like we have solved it above for strings in line-based protocols, by escaping any occurence of the special delimiter character within frames. That is, if you want to use 0x00 as a delimiter, you would encode a packet containing 0xde 0xad 0x00 0xbe 0xef as something like 0xde 0xad 0x01 0x02 0xbe 0xef, replacing the null byte with a magic sequence. This framing works, but is has one critical disadvantage: The length of the resulting escaped data is dependent on the raw data, and in the worst case twice as long. In a raw packet consisting entirely of null bytes, every byte must be escaped with two escape bytes. This means that in this case the packet length doubles, and in this particular case we're even less efficient than base-64.

Highly variable packet length is also bad since it makes it very hard to make any timing guarantees for our protocol.

9-bit framing

A framing mode sometimes used is to configure the UARTs to transmit 9-bit characters and to use the 9th bit to designate control characters. This works really well, and gives plenty of control characters to work with. The main problem with this is that a 9-bit serial interface is highly nonstandard and you need UARTs on both ends that actually support this mode. Another issue is that though more efficient than both delmitier-based and purely text-based protocols, it still incurs an extra about 10% of bandwidth overhead. This is not a lot if all you're sending is a little command every now and then, but if you're trying to push large amounts of data through your serial it's still bad.

COBS

Given the limitations of the two above-mentioned framing formats, we really want something better. The Serial Line Internet Protocol (SLIP) as well as the Point to Point Protocol (PPP), standardized in 1988 and 1994 respectively, both use escape sequences. This might come as a surprise, but humanity has actually still made significant technological progress on protocols for 8-bit serial interfaces until the turn of the millennium. In 1999, Consistent Overhead Byte Stuffing (COBS) (wiki) was published by a few researchers from Apple Computer and Stanford University. As a reaction on the bandwidth doubling problem present in PPP, COBS always has an overhead of a single byte, no matter what or how long a packet's content is.

COBS uses the null byte as a delimiter interleaves all the raw packet data and a run-length encoding of the non-zero portions of the raw packet. That is, it prepends the number of bytes until the first zero byte to the packet, plus one. Then it takes all the leading non-zero bytes of the packet, unmodified. Then, it again encodes the distance from the first zero to the second zero, plus one. And then it takes the second non-zero run of bytes unmodified. And so on. At the end, the packet is terminated with a zero byte.

The result of this scheme is that the encoded packet does not contain any zero bytes, as every zero byte has been replaced with the number of bytes until the next zero byte, plus one, and that can't be zero. Both formatter and parser each have to keep a counter running to keep track of the distances between zero bytes. The first byte of the packet initializes that counter and is dropped by the parser. After that, every encoded byte received results in one raw byte parsed.

While this might sound more complicated than the escaping explained above, the gains in predictability and efficiency are worth it. An implementation of encoder and decoder should each be about ten lines of C or two lines of Python. A minor asymmetry of the protocol is that while decoding can be done in-place, encoding either needs two passes or you need to scan forward for the next null byte.