1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
|
<!DOCTYPE html>
<html><head>
<meta charset="utf-8">
<title>How to talk to your microcontroller over serial | Home</title>
<meta name="description" content="">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="mobile-web-app-capable" content="yes">
<meta name="color-scheme" content="dark light">
<link rel="stylesheet" href="/style.css">
</head>
<body><nav>
<div class="internal">
<a href="/" title="Home">Home</a>
<a href="/blog/" title="Blog">Blog</a>
<a href="/projects/" title="Projects">Projects</a>
<a href="/about/" title="About">About</a>
</div>
<div class="search">
<div id="search"></div>
</div>
<div class="external">
<a href="https://git.jaseg.de/" title="cgit">cgit</a>
<a href="https://github.com/jaseg" title="Github">Github</a>
<a href="https://gitlab.com/neinseg" title="Gitlab">Gitlab</a>
<a href="https://chaos.social/@jaseg" title="Mastodon">Mastodon</a>
</span>
</nav>
<header>
<h1>How to talk to your microcontroller over serial</h1>
<ul class="breadcrumbs">
<li><a href="/">jaseg.de</a></li>
<li><a href="/blog/">Blog</a></li><li><a href="/blog/serial-protocols/">How to talk to your microcontroller over serial</a></li>
</ul>
<strong>2018-05-19</strong>
</header>
<main data-pagefind-body>
<div class="document">
<p>Scroll to the end for the <a class="reference internal" href="#conclusion">TL;DR</a>.</p>
<p>In this article I will give an overview on the protocols spoken on serial ports, highlighting common pitfalls. I will
summarize some points on how to design a serial protocol that is simple to implement and works reliably even under error
conditions.</p>
<p>If you have done low-level microcontroller firmware you will regularly have had to stuff some data up a serial port to
another microcontroller or to a computer. In the age of USB, an old-school serial port is still the simplest and
quickest way to get communication to a control computer up and running. Integrating a ten thousand-line USB stack into
your firmware and writing the necessary low-level drivers on the host side might take days. Poking a few registers to
set up your UART to talk to an external hardware USB to serial converter is a matter of minutes.</p>
<p>This simplicity is treacherous, though. Oftentimes, you start writing your serial protocol as needs arise. Things might
start harmless with something like <tt class="docutils literal">SET_LED ON\n</tt>, but as the code grows it is easy to end up in a hot mess of command
modes, protocol states that breaks under stress. The ways in which serial protocols break are manifold. The simplest one
is that at some point a character is mangled, leading to both ends of the conversation ending up in misaligned protocol
states. With a fragile protocol, you might end up in a state that is hard to recover from. In extreme cases, this leads
to code such as <a class="reference external" href="https://github.com/juhasch/pyBusPirateLite/blob/dece35f6e421d4f6a007d1db98d148e2f2126ebb/pyBusPirateLite/base.py#L113">this gem</a> performing some sort of arcane ritual to get back to some known state, and all just because
someone did not do their homework. Below we'll embark on a journey through the lands of protocol design, exploring the
facets of this deceptively simple problem.</p>
<div class="section" id="text-based-serial-protocols">
<h2>Text-based serial protocols</h2>
<p>The first serial protocol you've likely written is a human-readable, text-based one. Text-based protocols have the big
advantage that you can just print them on a terminal and you can immediately see what's happening. In most cases you can
even type out the protocol with your bare hands, meaning that you don't really need a debugging tool beyond a serial
console.</p>
<p>However, text-based protocols also have a number of disadvantages. Depending on your application, these might not matter
and in many cases a text-based protocol is the most sensible solution. But then, in some cases they might and it's good
to know when you hit one of them.</p>
<div class="section" id="problems">
<h3>Problems</h3>
<div class="section" id="low-information-density">
<h4>Low information density</h4>
<p>Generally, you won't be able to stuff much more than four or five bit of information down a serial port using a
single byte of a human-readable protocol. In many cases you will get much less. If you have 10 commands that are only
issued a couple times a second nobody cares that you spend maybe ten bytes per command on nice, verbose strings such as
<tt class="docutils literal">SET LED</tt>. But if you're trying to squeeze a half-kilobyte framebuffer down the line you might start to notice the
difference between hex and base-64 encoding, and a binary protocol might really be more up to the job.</p>
</div>
<div class="section" id="complex-parsing-code">
<h4>Complex parsing code</h4>
<p>On the computer side of thing, with the whole phalanx of an operating system, the standard library of your programming
language of choice and for all intents and purposes unlimted CPU and memory resources to spare you can easily parse
anything spoken on a serial port in real time, even at a blazing fast full Megabaud. The microcontroller side however is
an entirely different beast. On a small microcontroller, <a class="reference external" href="http://git.musl-libc.org/cgit/musl/tree/src/stdio/vfprintf.c">printf</a> alone will eat about half your flash. On most small
microcontrollers, you just won't get a regex library even though it would make parsing textual commands <em>so much
simpler</em>. Lacking these resources, you might end up hand-knitting a lot of low-level C code to do something seemingly
simple such as parsing <tt class="docutils literal">set_channel (13, <span class="pre">1.1333)\n</span></tt>. These issues have to be taken into account in the protocol design
from the beginning. For example, you don't really need matching parentheses, don't use them.</p>
</div>
<div class="section" id="fragile-protocol-state">
<h4>Fragile protocol state</h4>
<p>Say you have a <tt class="docutils literal">SET_DISPLAY</tt> command. Now say your display can display four lines of text. The obvious approach to this
is probably the <a class="reference external" href="https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">SMTP</a> or <a class="reference external" href="https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol">HTTP</a> way of sending <tt class="docutils literal">SET_DISPLAY\nThis is line 1\nThis is line 2\n\n</tt>. This would certainly
work, but it is very fragile. With this protocol, you're in trouble if at any point the terminating second newline
character gets mangled (say, someone unplugs the cable, or the control computer reboots, or a cosmic ray hits something
and <tt class="docutils literal">0x10 '\n'</tt> turns into <tt class="docutils literal">0x50 'P'</tt>).</p>
</div>
<div class="section" id="timeouts-don-t-work">
<h4>Timeouts don't work</h4>
<p>You might try to solve the problem of your protocol state machine tangling up with a timeout. "If I don't get a valid
command for more than 200ms I go back to default state." But consider the above example. Say, your control computer
sends a <tt class="docutils literal">SET_DISPLAY</tt> command every 100ms. If in one of them the state machine tangles up, the parser hangs since the
timeout is never hit, because a new line of text is arriving every 100ms.</p>
</div>
<div class="section" id="framing-is-hard">
<h4>Framing is hard</h4>
<p>You might also try to drop the second newline and using a convention such as <tt class="docutils literal">SET_DISPLAY</tt> is followed by two lines of
text, then commands resume.". This works as long as your display contents never look like commands. If you are only ever
displaying the same three messages on a character LCD that might work, but if you're displaying binary framebuffer
data you've lost.</p>
</div>
</div>
<div class="section" id="solutions">
<h3>Solutions</h3>
<div class="section" id="keep-the-state-machine-simple">
<h4>Keep the state machine simple</h4>
<p>In a text-based protocol, always use a single line of text to represent a single command. Don't do protocol states or
modes where you can toggle between different interpretations for a line. If you have to send human-readable text as part
of a command (such as <tt class="docutils literal">SET_DISPLAY</tt>), escape it so it doesn't contain any newlines.</p>
<p>This way, you keep your protocol state machine simple. If at any time your serial trips and flips a bit or looses a byte
your protocol will recover on the next newline character, returning to its base state.</p>
</div>
<div class="section" id="encode-numbers-in-hex-when-possible">
<h4>Encode numbers in hex when possible</h4>
<p>Printing a number in hexadecimal is a very tidy operation, even on the smalest 8-bit microcontrollers. In contrast,
printing decimal requires both division and remainder in a loop which might get annoyingly code- and time-intensive on
large numbers (say a 32-bit int) and small microcontrollers.</p>
<p>If you have to send fractional values, consider their precision. Instead of sending a 12 bit ADC result as a 32-bit
float formatted like <tt class="docutils literal">0.176513671875</tt> sending <tt class="docutils literal">0x2d3</tt> and dividing by 4096 on the host might be more sensible. If you
really have to communicate big floats and you can't take the overhead of including both <a class="reference external" href="http://git.musl-libc.org/cgit/musl/tree/src/stdio/vfprintf.c">printf</a> and <a class="reference external" href="http://git.musl-libc.org/cgit/musl/tree/src/stdio/vfscanf.c">scanf</a> you can
use hexadecimal floating point, which is basically <tt class="docutils literal"><span class="pre">hex((int)foo)</span> + "." + <span class="pre">hex((int)(65536*(foo</span> - <span class="pre">(int)foo)))</span></tt> for four
digits. You can also just hex-encode the binary <a class="reference external" href="https://en.wikipedia.org/wiki/IEEE_754">IEEE-754</a> representation of the float, sending <tt class="docutils literal"><span class="pre">hex(*(int</span> <span class="pre">*)&float)</span></tt>.
Most programming languages will have a <a class="reference external" href="https://docs.python.org/3.5/library/struct.html">simple, built-in means to parse this sort of thing</a>.</p>
</div>
<div class="section" id="escape-multiline-strings">
<h4>Escape multiline strings</h4>
<p>If you have to send arbitrary strings, escape special characters. This not only has the advantage of yielding a robust
protocol: It also ensures you can actually see everything that's going on when debugging. The string <tt class="docutils literal">"\r\n"</tt> is easy to
distinguish from <tt class="docutils literal">"\n"</tt> while your terminal emulator might not care.</p>
<p>The simplest encoding to use is the C-style backslash encoding. Host-side, most languages will have a <a class="reference external" href="https://docs.python.org/3.5/library/codecs.html#text-encodings">built-in means of
escaping a string like that</a>.</p>
</div>
</div>
<div class="section" id="encoding-binary-data">
<h3>Encoding binary data</h3>
<p>For binary data, hex and base-64 are the most common encodings. Since hex is simpler to implement I'd go with it unless
I really need the 30% bandwidth improvement base-64 brings.</p>
</div>
</div>
<div class="section" id="binary-serial-protocols">
<h2>Binary serial protocols</h2>
<p>In contrast to anything human-readable, binary protocols are generally more bandwidth-efficient and are easier to format
and parse. However, binary protocols come with their own version of the caveats we discussed for text-based protocols.</p>
<div class="section" id="the-framing-problem-in-binary-protocols">
<h3>The framing problem in binary protocols</h3>
<p>The most basic problems with binary protocols as with text-based ones is framing, i.e. splitting up the continuous
serial data stream into discrete packets. The issue is that it is that you have to somehow mark boundaries between
frames. The simplest way would be to use some special character to delimit frames, but then any 8-bit character you
could choose could also occur within a frame.</p>
<div class="section" id="slip-ppp-like-special-character-framing">
<h4>SLIP/PPP-like special character framing</h4>
<p>Some protocols solve this problem much like we have solved it above for strings in line-based protocols, by escaping any
occurence of the special delimiter character within frames. That is, if you want to use <tt class="docutils literal">0x00</tt> as a delimiter, you would
encode a packet containing <tt class="docutils literal">0xde 0xad 0x00 0xbe 0xef</tt> as something like <tt class="docutils literal">0xde 0xad 0x01 0x02 0xbe 0xef</tt>, replacing the
null byte with a magic sequence. This framing works, but is has one critical disadvantage: The length of the resulting
escaped data is dependent on the raw data, and in the worst case twice as long. In a raw packet consisting entirely of
null bytes, every byte must be escaped with two escape bytes. This means that in this case the packet length doubles,
and in this particular case we're even less efficient than base-64.</p>
<p>Highly variable packet length is also bad since it makes it very hard to make any timing guarantees for our protocol.</p>
</div>
<div class="section" id="bit-framing">
<h4>9-bit framing</h4>
<p>A framing mode sometimes used is to configure the UARTs to transmit 9-bit characters and to use the 9th bit to designate
control characters. This works really well, and gives plenty of control characters to work with. The main problem with
this is that a 9-bit serial interface is highly nonstandard and you need UARTs on both ends that actually support this
mode. Another issue is that though more efficient than both delmitier-based and purely text-based protocols, it still
incurs an extra about 10% of bandwidth overhead. This is not a lot if all you're sending is a little command every now
and then, but if you're trying to push large amounts of data through your serial it's still bad.</p>
</div>
<div class="section" id="cobs">
<h4>COBS</h4>
<p>Given the limitations of the two above-mentioned framing formats, we really want something better. The <a class="reference external" href="https://en.wikipedia.org/wiki/Serial_Line_Internet_Protocol">Serial Line
Internet Protocol (SLIP)</a> as well as the <a class="reference external" href="https://en.wikipedia.org/wiki/Point-to-Point_Protocol">Point to Point Protocol (PPP)</a>, standardized in 1988 and 1994 respectively,
both use escape sequences. This might come as a surprise, but humanity has actually still made significant technological
progress on protocols for 8-bit serial interfaces until the turn of the millennium. In 1999, <a class="reference external" href="http://www.stuartcheshire.org/papers/COBSforToN.pdf">Consistent Overhead Byte
Stuffing (COBS)</a> (<a class="reference external" href="https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing">wiki</a>) was published by a few
researchers from Apple Computer and Stanford University. As a reaction on the bandwidth doubling problem present in
<a class="reference external" href="https://en.wikipedia.org/wiki/Point-to-Point_Protocol">PPP</a>, COBS <em>always</em> has an overhead of a single byte, no matter what or how long a packet's content is.</p>
<p>COBS uses the null byte as a delimiter interleaves all the raw packet data and a <a class="reference external" href="https://en.wikipedia.org/wiki/Run-length_encoding">run-length encoding</a> of the non-zero
portions of the raw packet. That is, it prepends the number of bytes until the first zero byte to the packet, plus one.
Then it takes all the leading non-zero bytes of the packet, unmodified. Then, it again encodes the distance from the
first zero to the second zero, plus one. And then it takes the second non-zero run of bytes unmodified. And so on. At
the end, the packet is terminated with a zero byte.</p>
<p>The result of this scheme is that the encoded packet does not contain any zero bytes, as every zero byte has been
replaced with the number of bytes until the next zero byte, plus one, and that can't be zero. Both formatter and parser
each have to keep a counter running to keep track of the distances between zero bytes. The first byte of the packet
initializes that counter and is dropped by the parser. After that, every encoded byte received results in one raw byte
parsed.</p>
<p>While this might sound more complicated than the escaping explained above, the gains in predictability and efficiency
are worth it. An implementation of encoder and decoder should each be about ten lines of C or two lines of Python. A
minor asymmetry of the protocol is that while decoding can be done in-place, encoding either needs two passes or you
need to scan forward for the next null byte.</p>
</div>
</div>
<div class="section" id="state-machines-and-error-recovery">
<h3>State machines and error recovery</h3>
<p>In binary protocols even more than in textual ones it is tempting to build complex state machines triggering actions on
a sequence of protocol packets. Please resist that temptation. As with textual protocols keeping the protocol state to
the minimum possible allows for a self-synchronizing protocol. A serial protocol should be designed such that if due to
a dropped packet or two both ends will naturally re-synchronize within another packet or two. A simple way of doing that
is to always transmit one semantic command per packet and to design these commands in the most <a class="reference external" href="https://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning">idempotent</a> way possible.
For example, when filling a framebuffer piece by piece, include the offset in each piece instead of keeping track of it
on the receiving side.</p>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Here's your five-step guide to serial bliss:</p>
<ol class="arabic simple">
<li>Unless you have super-special requirements, always use the slowest you can get away with from 9600Bd, 115200Bd or
1MBd. 8N1 framing if you're talking to anything but another microcontroller on the same board. Using common values
like these makes it easier when you'll inevitably have to guess these at some point in the future ;)</li>
<li>If you're doing something simple and speed is not a particular concern, use a human-readable text-based protocol. Use
one command/reply per line, begin each line with some sort of command word and format numbers in hexadecimal. Bonus
points for the device replying to unknown commands with a human-readable status message and printing a brief protocol
overview on boot.</li>
<li>If you're doing something even slightly nontrivial or need moderate throughput (>1k commands per second or >20 byte of
data per command) use a COBS-based protocol. A good starting point is a <tt class="docutils literal">[target <span class="pre">MAC][command</span> <span class="pre">ID][command</span>
arguments]</tt> packet format for multidrop busses. For single-drop you may decide to drop the MAC address.</li>
<li>Always include some sort of "status" command that prints life stats such as VCC, temperature, serial framing errors
and uptime. You'll need some sort of ping command anyway and that command might as well do something useful.</li>
<li>If at all possible, keep your protocol context-free across packets/lines. That is, a certain command should always be
self-contained, and no command should change the meaning of the next packet/line/command that is sent. This is really
important to allow for self-synchronization. If you really need to break up something into multiple commands, say you
want to set a large framebuffer in pieces, do it in a <a class="reference external" href="https://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning">idempotent</a> way: Instead of sending something like <tt class="docutils literal">FRAMEBUFFER
<span class="pre">INCOMING:\n[byte</span> <span class="pre">0-16]\n[byte</span> <span class="pre">17-32]\n[...]\nEND</span> OF FRAME</tt> rather send <tt class="docutils literal">FRAMEBUFFER DATA FOR OFFSET 0: [byte
<span class="pre">0-16]\nFRAMEBUFFER</span> DATA FOR OFFSET 17: [byte <span class="pre">17-32]\n[...]\nSWAP</span> BUFFERS\n</tt>.</li>
</ol>
</div>
</div>
</main><footer>
Copyright © 2023 Jan Sebastian Götte
/ <a href="/about/">About</a>
/ <a href="/imprint/">Imprint</a>
</footer>
<script src="/pagefind/pagefind-ui.js"></script>
<script>
if(navigator.getEnvironmentIntegrity!==undefined)document.querySelector('body').innerHTML=`<h1>Your browser
contains Google DRM</h1>"Web Environment Integrity" is a Google euphemism for a DRM that is designed to
prevent ad-blocking, and which Google has forced into their browsers against widespread public opposition.
In support of an open web, this website does not function with this DRM. Please install a browser such
as <a href="https://www.mozilla.org/en-US/firefox/new/">Firefox</a> that respects your freedom and supports
ad blockers.`;
window.addEventListener('DOMContentLoaded', (event) => {
new PagefindUI({element: "#search", showSubResults: true});
});
</script>
</body>
</html>
|
