Binary Serial Protocols: COBS and MessagePack

Sending 23.51,1013.25,47,182\n over a serial port is a perfectly good idea, right up until it isn't. Text is easy to debug, every tool on Earth reads it, and at one sample per second it costs nothing. But push the sample rate up and two bills come due at once: the wire runs out of bandwidth, and the microcontroller burns cycles turning floats into strings. This post walks through the arithmetic, then through the two techniques that fix it: COBS for framing and MessagePack for encoding.

The arithmetic of a serial link

At 115200 baud with the usual 8N1 framing, every byte costs ten bits on the wire (start bit, eight data bits, stop bit). That gives you 11,520 bytes per second, total, assuming the transmitter never pauses. The CSV line above is 21 bytes, so the link tops out around 548 lines per second. The same record packed as three 32-bit floats and a 16-bit counter is 14 bytes, and with framing overhead about 16 on the wire: 720 frames per second, from the identical link, before you touch the baud rate.

The second bill is paid on the device. On an 8-bit AVR there is no hardware float support, so printf("%.2f") is a software decimal-conversion routine; just linking float support into avr-libc's printf costs roughly 1.5 KB of flash on a chip that may only have 32. Copying the same float into a buffer is a four-byte memcpy. On bigger Cortex-M parts the formatting itself is cheap enough, but the byte count on the wire still dominates, and that is a physics problem no faster CPU fixes.

The framing problem

So send raw structs. Except now you have a new problem: where does one frame end and the next begin? With text you had \n, a byte that never appears in the payload. In binary data every byte value from 0x00 to 0xFF is legitimate payload, so no delimiter is safe. Classic escape-based schemes (like SLIP) solve this by prefixing reserved bytes with an escape byte, which works but has an ugly worst case: payload full of reserved bytes doubles in size.

COBS: a delimiter that cannot appear in the data

COBS (Consistent Overhead Byte Stuffing, from a 1999 paper by Stuart Cheshire and Mary Baker) takes a different approach: transform the payload so that 0x00 never appears in it, then use 0x00 as the delimiter. The transform replaces every zero with a code byte that says how far away the next zero is. Encoding the four payload bytes 11 22 00 33:

payload:   11 22 00 33
encoded:   03 11 22 02 33 00
           └┬┘        └┬┘   └┬┘
   "zero in 3"  "zero in 2"  delimiter

The first code byte 03 means "two data bytes follow, then there was a zero." The zero itself is never transmitted; the code byte implies it. The trailing 00 is the frame delimiter, and it is guaranteed to be the only zero on the wire. The overhead is remarkably predictable: one byte for any frame up to 254 bytes, and at most one extra byte per 254 bytes after that. No payload can inflate it.

The property that makes COBS worth the trouble is resynchronization. If bytes get dropped mid-stream, a UART overruns, or the host attaches while the device is already talking, the receiver just discards input until the next 0x00 and is frame-aligned again. Corruption stays inside the frame it happened in. That is the whole reason firmware engineers reach for it on lossy or hot-pluggable links.

MessagePack: binary that describes itself

COBS answers "where does the frame end"; it says nothing about what the bytes inside mean. A raw packed struct is the smallest answer, but it is also a contract: firmware and host must agree on field order, type widths, endianness, and padding, forever, in two codebases. MessagePack is the middle ground: a binary encoding where every value carries a small type tag. Small integers cost one byte, a float32 costs five, an array header costs one. The telemetry record from earlier, sent as an array of three float32 values and a counter, is 19 bytes against 21 as CSV, and the gap widens as values gain digits: a float32 costs the same five bytes whether the reading is 47 or 47.03271, while text pays per character. Unlike the raw struct, it also still knows its own shape.

The MCU side is well served. In C, mpack writes MessagePack into a stack buffer with no malloc. On Arduino, ArduinoJson has shipped serializeMsgPack() since version 6. CircuitPython ships a msgpack module in the core. One caveat worth knowing: if your encoder emits float64 by default (Python's does), the size advantage over text mostly evaporates, so pin your floats to 32 bits when the precision allows it.

Decoding it on the host

This is the part that usually makes people keep CSV: the receiving side now needs a decoder. In Serial Studio that is a solved problem twice over. The frame parser has built-in templates for both of these formats (and for SLIP, TLV, MAVLink, and others): "COBS-encoded frames" removes the stuffing and hands each decoded byte to a channel, and "MessagePack data" decodes payloads either as a flat array or by routing map keys to named channels. You select the binary decoder, set 00 as the end delimiter for COBS, and you are plotting.

When your format is fully custom, you write the parser yourself, in Lua or JavaScript. Binary mode hands parse() an array of byte values rather than a string, and from there it is ordinary struct unpacking:

function parse(frame) {
  if (frame.length < 14)
    return [];

  const view = new DataView(Uint8Array.from(frame).buffer);
  return [
    view.getFloat32(0, true),   // temperature, little-endian
    view.getFloat32(4, true),   // pressure
    view.getFloat32(8, true),   // humidity
    view.getUint16(12, true)    // counter
  ];
}

The length guard is not decoration. Serial Studio probes a new parse function with tiny sample inputs before accepting it, and a live link will eventually deliver a truncated frame; in both cases reading a float from a too-short buffer is a runtime error, while returning an empty array simply skips the frame.

Throughput is the reason the host can keep up with what the wire delivers: the parsing hot path is benchmarked in CI, and a Serial Studio build fails if Lua numeric parsing drops below 256,000 frames per second on the build machine, with the pipeline as a whole gated at four times that. A 720-frame-per-second link is not going to be the bottleneck. Note that the binary decoder mode belongs to Serial Studio Pro; hex and Base64 decoding are in the free build.

Keep CSV when CSV is winning

None of this is a reason to abandon text for a sensor that reports once a second, and it is worth being honest about what you give up. A terminal shows CSV meaningfully with zero tooling. A logic analyzer decodes ASCII right on the capture, while a COBS frame is hex soup until you post-process it. A garbled text line is visibly garbled, while a flipped bit in a binary float becomes a plausible wrong number, which is why serious binary protocols also carry a checksum. The switch earns its complexity when rate times record size approaches what the link can carry, or when the microcontroller genuinely cannot afford to format text. Before that point, plain text is a feature, not a compromise.