Streaming Protocols

Streaming systems encode, transmit, and play back continuous audio and video. Protocol selection first distinguishes two link segments: ingest/contribution from the capture end to the media service, and distribution from the media service to the audience. The former may use SRT, while the latter uses HLS; they should not be treated as mutually exclusive competing options.

These are measurement dimensions, not fixed latency rankings. The same protocol can yield completely different end-to-end latencies under different encoders, buffers, and network conditions.

Three Types of Time: Start-up, Live Latency, and Buffer

Start-up time is the interval from when the user initiates playback to the display of the first frame; glass-to-glass latency is the time from capture in the real world to display for the viewer; buffer level is the amount of available media duration that the player has received but not yet played. VOD can start up in a very short time while maintaining a deep buffer; you cannot write "VOD is delayed by ten seconds" to mean "the buffer is ten seconds."

Live latency may include capture encoding, contribution, segment publish wait, manifest discovery, download, player safety margin, and display. If a certain stage waits for segments, the segment duration affects it, but the total latency is not always equal to segment duration × number of buffered segments. When calculating, avoid double-counting the same hold-back and buffer.

For example, if the clearly divided stages are 0.1, 0.2, 0.5, 0.1, 0.2, 1.0, and 0.1 seconds, the total is 2.2 seconds. Even if the publish wait is shortened to 0.1 second, with other stages unchanged, there is still 1.8 seconds; you cannot claim the system achieves 100 ms latency just because you see a 100 ms part.

ABR: Deducing Buffer Changes from Download Time

The server provides representations with different bitrates and resolutions; the client selects based on throughput, buffer, device decoding capability, etc. To switch smoothly, the media timeline, random access points, and encoding constraints need to cooperate; "same segment number" alone does not guarantee seamless switching.

Assume each segment is 2 seconds long, encoded at 4 Mbit/s, so the segment is approximately 8 Mbit; the link's actual available throughput is 2 Mbit/s, requiring 4 seconds to download. During these four seconds, the player consumes four seconds of old content but only replenishes two seconds of new content, resulting in a net loss of two seconds of buffer.

from fractions import Fraction as F

def next_buffer(buffer, duration, bitrate, throughput):
    download = duration * bitrate / throughput
    stall = max(F(0), download - buffer)
    return max(F(0), buffer - download) + duration, stall

buffer = F(6)
trace = []
for _ in range(3):
    buffer, stall = next_buffer(buffer, F(2), F(4), F(2))
    trace.append((buffer, stall))
assert trace == [(4, 0), (2, 0), (2, 2)]
assert next_buffer(F(6), F(2), F(2), F(2)) == (6, 0)
# 2 Mbit/s estimated throughput, 0.8 safety factor, select 1.4 from this candidate set.
ladder = [F('0.8'), F('1.4'), F('2.8'), F(5)]
assert max(rate for rate in ladder if rate <= F(2) * F('0.8')) == F('1.4')
print('Segment-by-segment buffer depletion, stall, and conservative selection model passed')

This is a teaching model with constant segment bitrate and no request overhead. In reality, VBR segment sizes vary, and throughput estimation is affected by sampling windows, cache hits, connection startup, and the fact that low-latency content may not yet be produced. You cannot assert that on-off downloads always overestimate or always underestimate bandwidth. dash.js throughput calculation

Throughput-based and buffer-based rules can be combined; actual players also consider abandoning slow requests, frame drops, and frequent switching. Stuttering may also come from missing segments at the origin, discontinuous timestamps, or insufficient terminal decoding, not just ABR algorithms. dash.js ABR

Download one segment, replenish two seconds, why does the buffer decrease?

Under default conditions, downloading three segments continuously reduces the buffer from 6 to 4 to 2, with a 2-second stall occurring for the third time. Lowering the bitrate mid-stream changes future download times but does not rewrite stalls that have already occurred. This turns a single calculation into a traceable cumulative process.

$$D=\frac{Tr}{c},stall=\max (0,D-B),B_{next}=\max (0,B-D)+T$$

HLS: Manifests Describe, Media Segments Contain Content

HLS uses UTF-8 playlists and supports MPEG-TS, fragmented MP4, and other media formats; it is not limited to .ts. Multi-variant playlists list available representations; media playlists describe specific media segments. Players can also open a single media playlist directly, without always going through the multi-variant layer first. HLS RFC 8216

Below is a complete short fMP4 VOD media playlist example; media files must be generated by compatible encoders/packers, and this text itself does not produce video.

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:2
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MAP:URI="init.mp4"
#EXTINF:2.000,
seg-000.m4s
#EXTINF:2.000,
seg-001.m4s
#EXT-X-ENDLIST

EXT-X-MAP specifies the initialization segment; EXTINF indicates the segment duration. The constraint on TARGETDURATION is that the integer value of segment EXTINF durations, rounded according to the specification, must not exceed it; it should not be broadly interpreted as the exact length of all segments. Media sequence numbers identify the segment sequence in the playlist and do not guarantee that the same sequence number across different variants can be swapped directly without timestamps.

Low Latency HLS: Partial Segments, Blocking Reload, and Preload Hints

LL-HLS allows already generated partial segments to be discovered and downloaded before the complete segment is finished. Blocking playlist reload allows the client to request future progress with _HLS_msn / _HLS_part; the server waits until the required content is available before returning, reducing the wait for repeatedly fetching old playlists. Apple LL-HLS

PRELOAD-HINT informs of potentially needed resources or ranges in advance, allowing the client to send requests first; it is not a guarantee that "the resource now exists completely," and plans may change. SERVER-CONTROL declares supported capabilities, and PART-INF describes partial segment duration parameters; these conventions require joint support from the origin, CDN, and player. Apple Blocking Reload Explanation

If caching ignores delivery directives, or if the proxy buffers the response until the complete segment ends, the origin may lose low-latency benefits even with fast publishing. Smaller parts increase request and scheduling overhead, and shallower buffers reduce jitter resistance, so smaller parts are not always better.

DASH and CMAF: Representations, Timelines, and Shareable Media

DASH's MPD organizes media using Period, AdaptationSet, and Representation. SegmentTemplate can locate segments by number or time. timescale=90000, duration=180000 means each segment is two seconds; timescale is not the frame rate.

The time model for dynamic MPD involves availability start time, client clock, time-shift window, and playback delay; minBufferTime is not a simple switch saying "the player must buffer exactly this many seconds." DASH-IF Time Model

Low Latency DASH can incrementally transmit CMAF chunks while a segment is still being generated. HTTP/1.1 can use chunked transfer coding, while HTTP/2/3 pass data according to their respective data frames; HTTP/1.1 chunked encoding cannot be listed as a necessary condition for all LL-DASH. DASH-IF Low Latency Guidelines

CMAF specifies media organization and constraints; it is not another video codec. When HLS and DASH are compatible in codec, tracks, segment boundaries, and encryption configuration, they can reuse media objects while using different manifests; if these conditions are not met, multiple package versions may still exist. "Using CMAF definitely halves caching" ignores actual encapsulation and terminal requirements.

CDN: Two Accounts for Distribution Copies and Origin Load

If ten thousand viewers each watch 3 Mbit/s, the edge-to-viewer requires approximately 30 Gbit/s of media throughput; high cache hit rates can reduce origin pulls but do not make these ten thousand viewer transmissions disappear. Assuming twenty edges each pull one continuous media stream for the same quality level from the origin, the ideal media origin pull is about 60 Mbit/s, not counting other quality levels, requests, warming, and failure retries.

Media segments that are complete and have non-reused URLs are suitable for longer caching; constantly changing live playlists require timely updates. If authentication parameters cause each user to form a different cache key, shared hits will decrease; ignoring authentication may lead to unauthorized access. Design so that the edge handles authorization and clearly defines which content can be shared, rather than simply adding long cache TTLs to all URLs.

RTMP(S) and SRT: Different Tools for Contribution

RTMP carries control and media messages via reliable byte streams; its chunk stream is used for message chunking and interleaving, not HLS media segments. RTMPS adds TLS protection. Flash browser playback exiting history does not mean RTMP cannot be used for any playback or modern contribution; Enhanced RTMP continues to extend codec and multi-track capabilities, depending on end-to-end support. Enhanced RTMP

SRT uses a transport mechanism over UDP; real-time mode balances recovery and timeliness through packet loss feedback/retransmission and timestamp delivery buffering. SRTO_RCVLATENCY and peer requirements participate in determining receive latency constraints; they are not the total latency from camera to display. Even if data arrives early, it may wait according to the delivery schedule. SRT Latency Mechanism

Increasing the latency budget gives more opportunities for retransmission but cannot fix persistent bandwidth shortages. If encoding is 8 Mbit/s but the link is only 6 Mbit/s long-term, simply increasing the buffer only delays failure; retransmission also consumes extra bandwidth. Caller, Listener, and Rendezvous are roles/methods for establishing connections, not guarantees that any NAT can be traversed.

Interactive and Game Streaming: Deadlines Come from the Entire Feedback Path

WebRTC is suitable for real-time media requiring two-way feedback; media servers can distribute according to subscriptions and layers; it can also be converted and combined with HTTP segment distribution on the same platform. Do not describe all WebRTC connections as direct connections or using the same congestion algorithm.

Game streaming like Sunshine/Moonlight also includes input contribution, game logic and rendering, encoding, downlink, decoding, and display; it evaluates the complete closed loop from operation to screen. 60 fps only indicates an ideal frame interval of about 16.7 ms; it does not prove that the entire closed loop must complete within one frame, nor can it assert that any retransmission is necessarily useless. FEC, retransmission, frame dropping, and frame pacing must be balanced within actual playback deadlines.

Content Encryption and Playback Authorization

HTTPS protects the client-to-service transmission; HLS AES-128 and other methods protect media objects; DRM also includes license and playback permission systems. The three are not the same switch. Using SAMPLE-AES does not guarantee that all terminals will support a certain DRM just by the tag; key format, encryption mode, and player capabilities must all match.

The fact that the key URL can be downloaded does not mean the business has completed authorization; URL signature expiration does not erase content the user has already received. When troubleshooting black screens, distinguish between manifest failure, media failure, initialization/codec support failure, and license failure, avoiding attributing all problems to network speed.

When selecting a solution, describe requirements using reproducible metrics: time to first display, source-to-playback latency, stall ratio, quality changes, endpoint resources, and per-viewer transmission cost. After retaining network and encoding conditions, measurements can support the next round of adjustments.