Binary Protocol Design

Binary protocol design must first define how the receiver obtains clear, bounded, and verifiable messages from bytes. Compact encoding is only one aspect; one must also specify field meanings, repetition handling, unknown values, length limits, state transitions, and error behaviors. Text can also have strict framing, and binary may be larger than some text; one cannot derive quality solely from form.

Framing: Length prefixes also require a complete contract

TCP does not preserve application write boundaries. Two send calls may appear in a single read, and one send may be completed by multiple read operations. UDP preserves datagram boundaries, but one still needs to determine if the application message is complete, whether fragmentation or multiple records are allowed, and if the receive buffer causes truncation.

The empty line in HTTP headers only delimits the end of the headers; it does not independently determine the length of the entire response body; SMTP DATA also relies on dot-stuffing escaping. One cannot treat these local mechanisms as universal framing that requires no other state.

The length field is not a trusted memory allocation instruction. The receiver should first validate the fixed header and length upper limit, then receive incrementally; for long-lived connections, one should also limit the number of incomplete messages, total buffer size, assembly time, and decompressed size. Limiting a single frame to 1 MiB still does not prevent ten thousand connections from each holding half a frame.

An instructional protocol for derivation

The following defines a small protocol used only to illustrate parsing boundaries; it is not an existing standard. It runs on a reliable, ordered byte stream, using a 12-byte fixed header and a TLV body; an empty body is allowed, and all multi-byte integers are big-endian.

The most significant bit of the TLV type indicates a critical field, and the lower 7 bits are the field ID. In this example, known ID 1 is a strict UTF-8 name, and ID 2 is a 4-byte uint32 quantity; known IDs are not allowed to repeat, other optional fields are skipped, and unknown critical fields are rejected. Missing known fields are allowed at the syntax level, but business operations may require them to be mandatory.

This "critical bit" is a rule actively defined by this protocol, not common to all TLV formats. Readers only need to know the length to skip optional values, but this does not imply understanding their semantics, nor does it guarantee that forwarding remains correct after discarding them.

Incremental parsing: Separate waiting, failure, and normal completion

The following parser limits body length and validates TLVs frame by frame. It receives chunks provided by the caller, requires the upper layer to feed bounded reads, and is responsible for connection-level budgets and timeouts; the example is not a complete server.

import struct
HEADER = struct.Struct('>2sBBII')
MAX_BODY = 4096

def parse_tlvs(body):
    pos, values = 0, {}
    while pos < len(body):
        if len(body) - pos < 3:
            raise ValueError('truncated TLV header')
        kind, size = struct.unpack_from('>BH', body, pos)
        pos += 3
        if size > len(body) - pos:
            raise ValueError('TLV exceeds body')
        value = body[pos:pos + size]
        pos += size
        field_id = kind & 0x7f
        if field_id not in (1, 2):
            if kind & 0x80:
                raise ValueError('unknown critical field')
            continue
        if field_id in values:
            raise ValueError('duplicate known field')
        if field_id == 1:
            values[1] = value.decode('utf-8', errors='strict')
        else:
            if size != 4:
                raise ValueError('quantity must be uint32')
            values[2] = struct.unpack('>I', value)[0]
    return values

class Decoder:
    def __init__(self):
        self.buffer = bytearray()

    def feed(self, chunk):
        self.buffer.extend(chunk)
        messages = []
        while len(self.buffer) >= HEADER.size:
            magic, version, flags, size, request_id = HEADER.unpack_from(self.buffer)
            if magic != b'KB' or version != 1 or flags != 0 or size > MAX_BODY:
                raise ValueError('invalid header')
            end = HEADER.size + size
            if len(self.buffer) < end:
                break
            fields = parse_tlvs(bytes(self.buffer[HEADER.size:end]))
            del self.buffer[:end]
            messages.append((request_id, fields))
        return messages

    def eof(self):
        if self.buffer:
            raise ValueError('truncated frame')

def tlv(kind, value):
    return struct.pack('>BH', kind, len(value)) + value

def frame(body, request_id=7):
    return HEADER.pack(b'KB', 1, 0, len(body), request_id) + body

body = tlv(1, '温度'.encode()) + tlv(2, struct.pack('>I', 300))
wire = frame(body)
assert len(body) == 16 and len(wire) == 28
expected = [(7, {1: '温度', 2: 300})]
# Exhaust every two-part read boundary, including empty reads and complete reads.
for split in range(len(wire) + 1):
    decoder = Decoder()
    assert decoder.feed(wire[:split]) + decoder.feed(wire[split:]) == expected
    decoder.eof()
# Two frames concatenated must also be consumed frame by frame.
assert Decoder().feed(wire + wire) == expected + expected
assert Decoder().feed(frame(body + tlv(3, b'optional'))) == expected

bad = [
    frame(body + tlv(0x83, b'critical')),
    frame(body + tlv(1, b'duplicate')),
    frame(tlv(1, b'\xff')),
    frame(tlv(2, b'\x01')),
    frame(b'\x01\x00\x04x'),
    HEADER.pack(b'KB', 1, 0, MAX_BODY + 1, 7),
    wire[:-1],
]
for candidate in bad:
    decoder = Decoder()
    try:
        decoder.feed(candidate)
        decoder.eof()
    except ValueError:
        pass
    else:
        raise AssertionError('invalid input accepted')
print('28 B message: all split boundaries, frame concatenation, optional fields, and 7 types of rejection passed')

After a parsing error, this example requires the caller to terminate decoding of that stream, rather than continuing to guess if the next KB is a frame header, because that byte sequence might also legally appear within the payload. If resynchronization in noise is indeed needed, synchronization, integrity, and recovery rules should be designed separately.

Production implementations must also consider the cost of moving buffers byte by byte, how to deliver messages parsed before an exception, memory ownership, and asynchronous cancellation. Zero-copy does not exempt one from lifecycle constraints: retaining a small view may prevent the entire large buffer from being released.

Is half a header a wait or an error?

At11 bytes, EOF turns waiting into truncation. At17, the first frame is complete even before the second arrives. Oversize mode rejects at12 bytes without waiting for4097 body bytes.

Integer Encoding: Protobuf varint and QUIC varint are different

Protobuf varint places the least significant group first, with uint64 taking up to 10 B; QUIC encodes in network byte order outside the length marker, with a maximum of 62 bits. QUIC's general integer encoding allows some non-shortest representations, but specific fields like frame type have separate shortest encoding requirements; it cannot be generalized as "all varints must be shortest". QUIC §16

def uvarint(n):
    if not 0 <= n < 1 << 64:
        raise ValueError('not uint64')
    result = bytearray()
    while n >= 128:
        result.append((n & 127) | 128)
        n >>= 7
    result.append(n)
    return bytes(result)

def quic_integer(n):
    for length, bits, prefix in [(1, 6, 0), (2, 14, 1), (4, 30, 2), (8, 62, 3)]:
        if 0 <= n < 1 << bits:
            return (n | (prefix << (8 * length - 2))).to_bytes(length, 'big')
    raise ValueError('not uint62')

assert uvarint(300).hex() == 'ac02'
assert quic_integer(300).hex() == '412c'
assert len(uvarint(127)) == 1 and len(uvarint(128)) == 2
assert len(quic_integer(63)) == 1 and len(quic_integer(64)) == 2
assert len(uvarint(2**64 - 1)) == 10
assert len(quic_integer(2**62 - 1)) == 8
# ZigZag maps signed numbers with small absolute values to small unsigned numbers.
zigzag = lambda n: 2*n if n >= 0 else -2*n-1
assert [zigzag(n) for n in [0, -1, 1, -2]] == [0, 1, 2, 3]
assert uvarint(zigzag(-2)) == b'\x03'
print('Two types of varints, boundaries, and ZigZag mapping passed')

ZigZag is a value mapping, followed by varint encoding. Protobuf's sint32/sint64 use it; negative numbers for int32/int64 follow another rule, where negatives may take 10 B. One cannot interchange declarations just because their wire type is 0.

Why do two varints encode300 differently?

Inspect63/64 where QUIC grows to2 bytes, then127/128 where Protobuf grows. These are different length rules. At300 compare group order in ac02 and412c.

Protobuf: Wire type gives boundaries, schema gives semantics

syntax = "proto3";
message Person {
  string name = 1;
  uint32 age = 2;
}

An encoding for name="Li", age=300 is 0a 02 4c 69 10 ac 02: the tag for field 1 is (1 << 3) | 2 = 0a, followed by length 2 and UTF-8; the tag for field 2 is 10, followed by varint ac 02. Field names do not appear on the wire, and age units and constraints are not interpreted by the tag itself.

Therefore, Protobuf is not a uniform three-segment TLV where each field explicitly carries an independent length. fixed32/fixed64 etc. use little-endian, which differs from the common big-endian of TCP/IP headers; variable-length integers should be understood according to their own bit-group rules. Protobuf Encoding

Retention of unknown fields, field presence, and JSON conversion behavior depend on the specific runtime and interface. Reading a binary message into an object and then manually copying only known fields may lose unknown content that could otherwise be retained; wire compatibility does not mean all intermediate processing is lossless. Proto3 Unknown Fields

Byte order, normalization, and integrity determine separately

New protocols should explicitly specify field width, sign, byte order, floating-point representation, text encoding, and units, rather than "just pick CPU native". Sending C structs directly introduces padding, alignment, and ABI differences; CPU allowing unaligned access does not imply the other end's language and hardware also allow it.

Signatures, hashes, and cache keys also require defining whether they bind to raw bytes or normalized semantics. Two equivalent objects do not imply their serialized bytes are unique. If a unique representation is needed, rules for field order, duplicate keys, floating-point NaN, and shortest number encoding should be explicit; CBOR's deterministic encoding is an example of explicitly writing out these requirements, rather than "all CBOR is naturally normalized". CBOR Deterministic Encoding

CRC or checksums mainly detect unexpected corruption under applicable failure models; they cannot prove the message comes from a trusted peer; authentication requires MAC, AEAD, or signatures and key trust. Compression also requires specifying decompression limits; a small original frame does not mean the resource overhead after expansion is small.

The criterion for design completion is that failure inputs also have deterministic behavior

Prioritize testing truncation, concatenation, boundary lengths, unknown critical fields, duplicate values, and cross-language vectors, then proceed to fuzzing and resource budget testing. Fault tolerance should be explicitly defined interoperable behavior, not each implementation guessing to accept different inputs. Maintaining Robust Protocols