Binary Serial Protocols: COBS & MessagePack

Sending 23.51,1013.25,47,182\n over a serial port is a perfectly good idea, right up until it isn't. Text is easy to debug, every tool on Earth reads it, and at one sample per second it costs nothing. But push the sample rate up and two bills come due at once: the wire runs out of bandwidth, and the microcontroller burns cycles turning floats into strings. This post walks through the arithmetic, then through the two techniques that fix it: COBS for framing and MessagePack for encoding.

Key takeaways

• The same 115200-baud link carries 548 CSV lines per second or 720 binary frames, no baud rate change required.
• COBS removes every 0x00 from the payload so it can be the frame delimiter, with bounded overhead of one byte per 254.
• MessagePack tags each value with its type, so the wire format still describes itself without CSV's per-character cost.
• Serial Studio ships parser templates for both formats, so plotting binary telemetry doesn't require writing a decoder from scratch.

The arithmetic of a serial link

At 115200 baud with the usual 8N1 framing, every byte costs ten bits on the wire (start bit, eight data bits, stop bit). That gives you 11,520 bytes per second, total, assuming the transmitter never pauses. The CSV line above is 21 bytes, so the link tops out around 548 lines per second. The same record packed as three 32-bit floats and a 16-bit counter is 14 bytes, and with framing overhead about 16 on the wire: 720 frames per second, from the identical link, before you touch the baud rate.

The second bill is paid on the device. On an 8-bit AVR there is no hardware float support, so printf("%.2f") is a software decimal-conversion routine; just linking float support into avr-libc's printf costs roughly 1.5 KB of flash on a chip that may only have 32. Copying the same float into a buffer is a four-byte memcpy. On bigger Cortex-M parts the formatting itself is cheap enough, but the byte count on the wire still dominates, and that is a physics problem no faster CPU fixes.

The framing problem

So send raw structs. Except now you have a new problem: where does one frame end and the next begin? With text you had \n, a byte that never appears in the payload. In binary data every byte value from 0x00 to 0xFF is legitimate payload, so no delimiter is safe. And the delimiter was doing a second job all along: it is how a receiver that attaches mid-stream, or loses a byte to an overrun, ever finds the next frame boundary. Give that up and a single corrupted byte can shift every frame after it. Classic escape-based schemes (like SLIP) solve this by prefixing reserved bytes with an escape byte, which works but has an ugly worst case: payload full of reserved bytes doubles in size.

COBS: a delimiter that cannot appear in the data

COBS (Consistent Overhead Byte Stuffing, from a 1999 paper by Stuart Cheshire and Mary Baker) takes a different approach: transform the payload so that 0x00 never appears in it, then use 0x00 as the delimiter. The transform replaces every zero with a code byte that says how far away the next zero is. Encoding the four payload bytes 11 22 00 33:

payload:   11 22 00 33
encoded:   03 11 22 02 33 00
           └┬┘        └┬┘   └┬┘
   "zero in 3"  "zero in 2"  delimiter

The first code byte 03 means "two data bytes follow, then there was a zero." The zero itself is never transmitted; the code byte implies it. The trailing 00 is the frame delimiter, and it is guaranteed to be the only zero on the wire. The overhead is remarkably predictable: one byte for any frame up to 254 bytes, and at most one extra byte per 254 bytes after that. No payload can inflate it.

packet-beta
title COBS encoding of payload 11 22 00 33 on the wire
0-7: "0x03 code, zero in 3"
8-15: "0x11 data"
16-23: "0x22 data"
24-31: "0x02 code, zero in 2"
32-39: "0x33 data"
40-47: "0x00 delimiter"

The property that makes COBS worth the trouble is resynchronization. If bytes get dropped mid-stream, a UART overruns, or the host attaches while the device is already talking, the receiver just discards input until the next 0x00 and is frame-aligned again. Corruption stays inside the frame it happened in. That is the whole reason firmware engineers reach for it on lossy or hot-pluggable links.

MessagePack: binary that describes itself

COBS answers "where does the frame end"; it says nothing about what the bytes inside mean. A raw packed struct is the smallest answer, but it is also a contract: firmware and host must agree on field order, type widths, endianness, and padding, forever, in two codebases. MessagePack is the middle ground: a binary encoding where every value carries a small type tag. Small integers cost one byte, a float32 costs five, an array header costs one. The telemetry record from earlier, sent as an array of three float32 values and a counter pinned to 16 bits, is 19 bytes against 21 as CSV, and the gap widens as values gain digits: a float32 costs the same five bytes whether the reading is 47 or 47.03271, while text pays per character. Unlike the raw struct, it also still knows its own shape.

The MCU side is well served. In C, mpack writes MessagePack into a stack buffer with no malloc. On Arduino, ArduinoJson has shipped serializeMsgPack() since version 6.0, and it is still there in v7. CircuitPython ships a msgpack module in the core. One caveat worth knowing: if your encoder emits float64 by default (Python's does), the size advantage over text mostly evaporates, so pin your floats to 32 bits when the precision allows it.

What the encoder costs on the device

Wire size is only one of three budgets. The other two are flash and RAM, and on small parts they decide the question before bandwidth gets a vote. A Cortex-M0 with 8 KB of SRAM, or an AVR with 2, does not evaluate serialization libraries the way a server does.

The packed struct sits at one extreme: it costs nothing, because it never serializes at all. The compiler fixed the layout at build time, the struct in memory is the wire format, and transmission is a memcpy into the output buffer. Zero heap, zero encoder flash, fully deterministic. Every byte of that saving is paid for in contract: field order, type widths, endianness, and padding are now frozen into two codebases at once.

Runtime encoders spend resources to soften that contract, and how they spend RAM matters more than how much. The encoders worth using on a microcontroller work like mpack does: the caller hands over a buffer, usually a static or stack array sized at compile time, and the library refuses to allocate. The encoders to avoid are the ones that malloc per message or per node. On a chip with a few kilobytes of SRAM, the heap is not a convenience, it is a liability: allocate and free variable-sized blocks long enough and the free space fragments into pieces too small to use, which is why safety-oriented firmware standards (MISRA C among them) simply prohibit dynamic allocation after initialization. A telemetry encoder that allocates is a device that fails at hour forty of a soak test instead of at your desk.

For the four-field record this post has been carrying around, the budgets stack up like this:

Framing has a quieter entry in the same ledger. COBS's bounded overhead, one byte per 254, means the worst-case encoded size is known at compile time, so the transmit buffer can be a static array barely larger than the frame. An escape-based scheme like SLIP has a worst case that doubles the payload, so the honest static buffer is twice the frame size, or the code paths that "know" the payload never contains reserved bytes are one firmware revision away from being wrong.

Decoding it on the host

This is the part that usually makes people keep CSV: the receiving side now needs a decoder. In Serial Studio that is a solved problem twice over. The frame parser has built-in templates for both of these formats (and for SLIP, TLV, MAVLink, and others): "COBS-encoded frames" removes the stuffing and hands each decoded byte to a channel, and "MessagePack data" decodes payloads either as a flat array or by routing map keys to named channels. You select the binary decoder, set 00 as the end delimiter for COBS, and you are plotting.

When your format is fully custom, you write the parser yourself, in Lua or JavaScript. Binary mode hands parse() an array of byte values rather than a string, and from there it is ordinary struct unpacking:

function parse(frame) {
  if (frame.length < 14)
    return [];

  const view = new DataView(Uint8Array.from(frame).buffer);
  return [
    view.getFloat32(0, true),   // temperature, little-endian
    view.getFloat32(4, true),   // pressure
    view.getFloat32(8, true),   // humidity
    view.getUint16(12, true)    // counter
  ];
}

The length guard is not decoration. Serial Studio probes a new parse function with tiny sample inputs before accepting it, and a live link will eventually deliver a truncated frame; in both cases reading a float from a too-short buffer is a runtime error, while returning an empty array simply skips the frame.

Throughput is the reason the host can keep up with what the wire delivers: the parsing hot path is benchmarked in CI, and a Serial Studio build fails if Lua numeric parsing drops below 256,000 frames per second on the build machine, while the lighter stages, raw byte extraction with no scripting and the built-in C++ parser, are gated at four times that. A 720-frame-per-second link is not going to be the bottleneck.

Note that the binary decoder mode belongs to Serial Studio Pro; hex and Base64 decoding are in the free build.

Keep CSV when CSV is winning

None of this is a reason to abandon text for a sensor that reports once a second, and it is worth being honest about what you give up. A terminal shows CSV meaningfully with zero tooling. A logic analyzer decodes ASCII right on the capture, while a COBS frame is hex soup until you post-process it. A garbled text line is visibly garbled, while a flipped bit in a binary float becomes a plausible wrong number, which is why serious binary protocols also carry a checksum; for how a CRC actually catches corruption, see Checksums and CRC, Explained. The switch earns its complexity when rate times record size approaches what the link can carry, or when the microcontroller genuinely cannot afford to format text. Before that point, plain text is a feature, not a compromise.