Binary Protocol Design

Binary protocols are more compact than text protocols, but every design decision—framing (length-prefixed vs delimiter), integer encoding (varint vs fixed), extensibility (TLV allows parsers to skip unknown fields)—affects the protocol’s bandwidth efficiency, parsing complexity, and forward compatibility. Protobuf’s wire format is a classic reference for these decisions.

Overview

Binary protocols are more compact than text protocols, but require careful design of field encoding and extensibility. Core decisions: framing method (length-prefixed vs delimiter), integer encoding (varint vs fixed-width), extensibility (TLV allows parsers to skip unknown fields). Protobuf (Google, 2008) is widely used for microservice communication and configuration serialization; its wire format design—field number + wire type tag + varint/length-delimited data—is an excellent example of binary protocol design.

Framing

Length-prefixed:
  [4-byte len (big-endian)][payload]
  Pros: Simple, read len → allocate buffer → read payload
  Cons: Requires knowing the length in advance

Delimiter-based:
  [payload][CR LF CR LF]  ← HTTP header boundary
  [payload][CR LF . CR LF] ← SMTP body end
  → Requires escaping when delimiter appears in payload

TLV (Type-Length-Value)

Extensibility: if the parser doesn't recognize the type → skip by length → forward compatible

[Type (1-2B)] [Length (1-4B)] [Value (length bytes)]

Integer Encoding

Fixed: 2/4/8B big-endian (IP, TCP, DNS)

varint (protobuf, QUIC):
  Highest bit of each byte = "more bytes follow"
  Example: 300 = 0x12C → 1010 1100 0000 0010 = AC 02 (2B)

ZigZag (signed int):
  sint32: 0→0, -1→1, 1→2, -2→3 → smaller encoding for small absolute values

Protobuf Wire Format

field = (field_number << 3) | wire_type

wire_type: 0=varint, 2=length-delimited

message Person {
  string name = 1;  → tag: 0x0A (1<<3 | 2), len, UTF-8 bytes
  int32  age  = 2;  → tag: 0x10 (2<<3 | 0), varint
}

Byte Order

Big-endian (network byte order): TCP/IP headers → htons() / htonl()
Little-endian (host): x86/ARM → new protocols use LE (CPU native, fewer conversions)

References

protobuf: protobuf.dev/programming-guides/encoding
QUIC: RFC 9000 (varint encoding in Appendix A)

Keywords: framing, TLV, varint, protobuf, big-endian, little-endian, ZigZag