Streaming Protocols

HLS splits video into ts segments for HTTP distribution (simple, CDN-friendly), while DASH is its codec-agnostic version. WebRTC takes a different path—low latency, P2P, suitable for interactive scenarios. The three protocols make different trade-offs between latency and scalability.

Overview

The essence of streaming is a trade-off between latency vs. scale. On one end are VOD (Video on Demand) and large-scale live broadcasts: media is chopped into 2-10s segments, distributed via standard HTTP, and pushed to tens of millions of viewers using CDN caching, at the cost of several seconds to tens of seconds of latency. On the other end are real-time interactions (video conferencing, cloud gaming): frames are pushed directly using RTP/UDP, squeezing latency down to tens of milliseconds, but each connection is stateful and difficult to scale using CDNs.

Understanding where each protocol sits on this spectrum is more important than memorizing their fields:

Why have HTTP-based segmented protocols (HLS/DASH) become the mainstream for large-scale live and VOD? Because they reduce "streaming" to "a series of cacheable small files": CDNs don't need to understand video; standard HTTP caching works; penetrating firewalls/NAT costs nothing (it's just GET); client logic is simple. The cost is latency—latency is essentially segment duration × number of buffered segments, which is the core contradiction that LL-HLS/LL-DASH aim to solve.

Adaptive Bitrate (ABR) — The Heart of Segmented Streaming

Segments exist not for "convenient downloading," but to seamlessly switch quality at segment boundaries. The server encodes an "encoding ladder" for the same content, where each stream is chopped into time-aligned segments; after downloading one or two segments, the client re-evaluates the network and decides which bitrate tier to fetch for the next segment.

flowchart LR
    subgraph ladder["Bitrate Ladder — Time-Aligned Segments"]
        A["1080p @ 5 Mbps"]
        B["720p @ 2.8 Mbps"]
        C["480p @ 1.4 Mbps"]
        D["360p @ 0.8 Mbps"]
    end
    P["🎬 <b>Player ABR</b><br><small>Re-evaluates network every 1-2 segments</small>"] ==>|"Ample bandwidth, deep buffer"| A
    P -.->|"Network degrades → Drop tier<br>at segment boundary"| C
    classDef player fill:#4493f826,stroke:#4493f8,stroke-width:2.5px
    classDef hi fill:#3fb9501f,stroke:#3fb950,stroke-width:2px
    classDef lo fill:#d299221f,stroke:#d29922,stroke-width:2px
    classDef rung fill:#64748b14,stroke:#64748b
    class P player
    class A hi
    class C lo
    class B,D rung

Segments of all tiers for segment N cover the same time interval [t, t+6s). Tier switching only happens at boundaries, allowing the decoder to衔接 seamlessly.

There are two major schools of ABR decision-making:

1. Throughput-based: Estimates bandwidth using the download speed of recent segments and selects a tier with a bitrate lower than the estimated bandwidth multiplied by a safety factor. The problem is that segmented downloads are on-off—idle after finishing a segment. Naive speed tests systematically underestimate bandwidth.
2. Buffer-based (e.g., BOLA): Models "which tier to choose" as a function of buffer level—dare to pick high bitrates when the buffer is deep, and conservatively drop tiers when the buffer is shallow to prevent stuttering.

Production implementations (dash.js / hls.js) are mostly hybrids of both, with special handling during startup (low tier for quick start). ABR is the true determinant of streaming experience: stuttering, sudden quality changes, and slow startup are almost always ABR strategy issues, not protocol issues.

HLS

Released by Apple in 2009, standardized in RFC 8216. Pure text .m3u8 playlist + media segments (early MPEG-TS .ts, modern uses fMP4/CMAF). Two-level playlist structure:

Master playlist — lists only the variants:
  #EXTM3U
  #EXT-X-STREAM-INF:BANDWIDTH=5000000,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=1920x1080
  high.m3u8
  #EXT-X-STREAM-INF:BANDWIDTH=2000000,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=1280x720
  medium.m3u8

Media playlist — list of segments for a specific variant:
  #EXTM3U
  #EXT-X-VERSION:7
  #EXT-X-TARGETDURATION:6          ← Max segment duration (rounded)
  #EXT-X-MEDIA-SEQUENCE:0          ← Sequence number of the first segment (increments for live)
  #EXT-X-PLAYLIST-TYPE:VOD         ← VOD/EVENT; omitted for live
  #EXTINF:6.000,
  segment-000.ts
  #EXTINF:6.000,
  segment-001.ts
  #EXT-X-ENDLIST                   ← VOD specific: marks end of stream

Playback flow: Fetch master → Select variant based on bandwidth/resolution → Fetch its media playlist → Sequentially fetch segments → Re-evaluate bandwidth and potentially switch tiers every 1-2 segments.

The difference between VOD and LIVE lies entirely in how the playlist changes:

• VOD: Playlist is provided all at once, includes EXT-X-ENDLIST, allowing clients to seek freely.
• EVENT: Segments are only appended to the end (rewatching full history, e.g., concerts).
• LIVE: Sliding window—new segments are appended, old ones removed, no ENDLIST; clients must periodically (approx. every target duration) re-fetch the playlist to get new segments. This "polling the playlist" is one of the main reasons for HLS's high latency.

Encryption: #EXT-X-KEY:METHOD=AES-128,URI="key.bin",IV=0x..., segments encrypted with AES-128-CBC; SAMPLE-AES used with FairPlay/Widevine for DRM.

LL-HLS (Low-Latency HLS)

Two actions reduce latency from 10-30s to 1-2s: chop segments further into partial segments (approx. 200-500ms) and publish them as they are encoded; eliminate polling latency using block-based playlist reloading—the client requests playlist?_HLS_msn=5&_HLS_part=2, the server holds the request and only returns it once part 5.2 is actually generated, eliminating idle polling.

#EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=1.0
#EXT-X-PART-INF:PART-TARGET=0.33
...
#EXT-X-PART:DURATION=0.33,URI="seg5.0.ts"
#EXT-X-PART:DURATION=0.33,URI="seg5.1.ts"
#EXT-X-PRELOAD-HINT:TYPE=PART,URI="seg5.2.ts"   ← Hint for next part, client can request early

MPEG-DASH

ISO/IEC 23009-1, vendor and codec agnostic (HLS historically tied to Apple ecosystem, DASH was codec-agnostic from the start). The playlist is an XML-based MPD (Media Presentation Description):

<MPD type="dynamic" minBufferTime="PT1.5S"            ← dynamic=live, static=VOD
     profiles="urn:mpeg:dash:profile:isoff-live:2011">
  <Period>
    <AdaptationSet mimeType="video/mp4" codecs="avc1.640028">
      <Representation id="720p" bandwidth="2000000" width="1280" height="720">
        <SegmentTemplate timescale="90000" duration="180000"   ← 180000/90000 = 2s
                         media="seg-$Number$.m4s" initialization="init.mp4"/>
      </Representation>
      <Representation id="1080p" bandwidth="5000000" width="1920" height="1080">...</Representation>
    </AdaptationSet>
  </Period>
</MPD>

SegmentTemplate uses $Number$ (incrementing index) or $Time$ (timestamp) to address segments, eliminating the need to list segments individually in the playlist—this is where DASH is more compact than HLS media playlists. Low-Latency DASH (LL-DASH) uses CMAF chunks + HTTP chunked transfer: a segment is divided into chunks internally; as soon as one chunk is encoded, it is pushed to the client via chunked transfer, without waiting for the entire segment to finish encoding.

Significance of CMAF: HLS and DASH have long used their own segment formats, requiring CDNs to store two copies. CMAF (Common Media Application Format, fMP4) allows both to share the same media segments, differing only in playlists (.m3u8 vs .mpd) → halving storage/caching requirements. This is the current direction of convergence.

RTMP/RTMPS — The Living Fossil of Streaming

Designed by Adobe for Flash, running on TCP port 1935. Flash died in 2020, and RTMP as a playback protocol is dead, but as a streaming ingest protocol, it still reigns supreme: almost all encoders and platforms support it by default.

flowchart LR
    OBS["🎥 OBS / Mobile App<br><small>Streamer side, 1 stream</small>"] -->|"RTMP(S) ingest"| T["⚙️ <b>Transcoding Service</b><br><small>Outputs multiple bitrates<br>Main source of latency and cost</small>"]
    T -->|"Packaging"| PKG["HLS / DASH"]
    PKG --> CDN["🌍 CDN"] ==> V["Tens of Millions of Viewers"]
    classDef src fill:#64748b1f,stroke:#64748b,stroke-width:2px
    classDef hot fill:#d2992226,stroke:#d29922,stroke-width:2.5px
    classDef dist fill:#4493f81f,stroke:#4493f8,stroke-width:2px
    class OBS src
    class T hot
    class PKG,CDN,V dist

Transcoding is the main source of latency and cost in the entire chain. Key protocol points:

• Handshake: C0/C1/C2 ↔ S0/S1/S2, exchanging version + random timestamps, echoing back for verification.
• Chunking: Chunk stream multiplexes audio/video/metadata into a single TCP connection (default 128B chunks).
• Metadata/RPC: AMF (Action Message Format) encodes connect / publish / onMetaData, etc.
• RTMPS = RTMP over TLS, the encrypted form for pushing streams to platforms.

It still dominates ingest due to ecosystem inertia rather than technical superiority. SRT and WebRTC are encroaching on this position.

WebRTC — Sub-second Real-time

Born for real-time browser communication, it is the de facto standard for video conferencing and interactive live streaming, with end-to-end latency under 500ms. It is not a single protocol, but a stack:

• Signaling (out-of-band, self-implemented): SDP offer/answer exchange for codec capabilities and ICE candidates.
• NAT Traversal: ICE framework—STUN asks the public network "what is my external address?" (direct connection works in most cases), TURN acts as a relay fallback if direct connection fails (incurs bandwidth costs).
• Encryption: DTLS handshake negotiates keys → SRTP encrypts media. Mandatory encryption, no plaintext mode.
• Media: RTP/RTCP for audio/video; congestion control uses GCC (Google Congestion Control), based on latency gradients rather than packet loss.

Why WebRTC is hard to scale like HLS: Each connection is stateful (ICE/DTLS/SRTP context), not a cacheable file. Scaling relies on server-side topology:

flowchart TB
    subgraph mesh ["😵 P2P mesh — N² streams, suitable only for 3-4 people"]
        m1((A)) --- m2((B)) --- m3((C)) --- m1
    end
    subgraph sfu ["✅ SFU — Mainstream Solution"]
        s1((A)) -->|"Upstream 1 stream"| S["<b>SFU</b><br><small>Forwards only, no decoding</small>"]
        s2((B)) --> S
        S ==>|"On-demand distribution"| s3((C))
    end
    subgraph mcu ["💸 MCU — Extremely CPU expensive, gradually deprecated"]
        u1((A)) --> M["<b>MCU</b><br><small>Decodes + composes into 1 stream</small>"]
        u2((B)) --> M
        M -->|"Mixed stream 1"| u3((C))
    end
    classDef good fill:#3fb95026,stroke:#3fb950,stroke-width:2.5px
    classDef bad fill:#f851491f,stroke:#f85149,stroke-width:2px
    classDef peer fill:#64748b1f,stroke:#64748b
    class S good
    class M bad
    class m1,m2,m3,s1,s2,s3,u1,u2,u3 peer

See WebRTC for details.

SRT — Reliable Streaming over Unreliable Networks

Open source by Haivision, based on UDT (UDP-based Data Transfer), positioned as reliable low-latency transmission across public networks/WANs, replacing satellite and RTMP for event contribution (contribution).

Core mechanism is ARQ (Selective Repeat) + a configurable latency buffer: The sender maintains a retransmission window, the receiver detects packet loss and sends NAKs, retransmitting only lost packets; the latency buffer (e.g., 120ms) is the time budget reserved for retransmission—trading "a bit more latency" for "quality on lossy links". This latency knob is the essence of SRT: the worse the link, the larger you set it.

• Encryption: AES-128/256.
• Connection Modes: Caller (active) / Listener (passive) / Rendezvous (bidirectional simultaneous initiation, for NAT traversal).

Compared to RTMP (TCP, where packet loss causes head-of-line blocking and latency spikes), SRT performs much more stably on lossy links, which is its fundamental advantage for ingest/contribution.

Moonlight/Sunshine — Cloud Gaming / Streaming

Open source implementation of NVIDIA GameStream (Sunshine server + Moonlight client), treating "real-time rendered game frames" as ultra-low-latency video streams.

flowchart LR
    G["🎮 Game Rendering<br><small>5-10ms</small>"] --> E["GPU Encoding<br><small>NVENC/AMF/VAAPI<br>5-10ms</small>"]
    E ==>|"RTP over UDP<br>+ FEC"| D["Client Decoding<br><small>5-10ms</small>"]
    D --> V["🖥️ Display VSYNC<br><small>5-15ms</small>"]
    V -.->|"Controller/Keyboard input, Haptic feedback"| G
    classDef host fill:#d299221f,stroke:#d29922,stroke-width:2px
    classDef net fill:#4493f826,stroke:#4493f8,stroke-width:2px
    classDef client fill:#3fb9501f,stroke:#3fb950,stroke-width:2px
    class G,E host
    class D,V client

Ideal end-to-end latency on LAN is 20-40ms—less than one and a half frames.

Why use UDP + FEC instead of retransmission: The deadline for real-time frames is only one frame (60fps → 16ms). Waiting for a retransmission round-trip will definitely miss the display window; a retransmitted frame is useless by then. So it's better to use Forward Error Correction (FEC, Reed-Solomon): send redundant packets, recover minor losses locally with zero round-trip. Combined with frame pacing to smooth display and avoid jitter.

Protocol Selection

References

• HLS: developer.apple.com/streaming (includes LL-HLS spec)
• DASH: dashif.org · ISO/IEC 23009-1
• WebRTC: webrtc.org · RFC 8825 series
• SRT: github.com/Haivision/srt
• Sunshine: github.com/LizardByte/Sunshine
• Reference Implementations: hls.js · dash.js (Industrial standard implementations of ABR algorithms, worth reading the source code)

Keywords: HLS, LL-HLS, DASH, CMAF, ABR, BOLA, RTMP, WebRTC, SFU, ICE, SRT, ARQ, Moonlight, Sunshine, NVENC, FEC, adaptive bitrate