Skip to content

My reading

Your bookmarks, everywhere Choose an account to sync across devices

This browser has local bookmarks. Merge them into this account? Duplicates are skipped and local data is kept.

Permanently delete your site account and cloud bookmarks, and sign out all devices. Existing local bookmarks stay. This cannot be undone.

Progress saved in this browser

Start a learning journey

Add a series from the atlas to track your reading progress.

Browse all series

Browse all series ↗

Your next good read

Save articles with the bookmark beside their title, then revisit them here.

This browser · Last 50 articles

Clear local reading history and positions?

Pick up where you left off

Articles appear here after a little reading, ready for your next visit.

Reading history could not be saved. Check browser storage settings.

liz's doc / Tags / #Information theory

What would you like to learn from the docs?

Find answers in this site’s docs, with links to the sources.

Ready to search

Sources 0

AI can make mistakes; check the source Open as a page ↗

Ask the docs

中文 English 日本語

Directory

Explorer

Programming

Models & agents 16

Model Foundations

Tokens, probabilities and sampling Attention, feed-forward networks and MoE Inference, KV cache and memory Inference-time compute, candidates and verification

Requests and Evidence

Prompts and output contracts Context engineering Retrieval, RAG and evidence

Agent Execution Systems

Agent loops and executors Tool interfaces and MCP Skills and reusable task methods Memory, state and recovery Authority, trust and tool boundaries Multi-agent orchestration and integration

Evaluation and Production

Evaluation, trials and observability Cost, capacity and reliability Release validation, canaries and recovery

Compilers 20

Lexical Analysis

Regular Expressions and Automata Lexer Design

Syntax Analysis

Recursive Descent and LL Analysis LR Parsing AST Design and Error Recovery

Semantic Analysis

Symbol Tables and Scopes Type Systems

Intermediate Representation

SSA Form LLVM IR in Practice

Optimization

Data Flow Analysis Classic Optimizations Register Allocation

Code Generation

Instruction Selection Object Code and ABI

Runtime

Garbage Collection Linking and Loading

Compilation Technology Applications

JIT Compilation DSL Design Compile-Time Metaprogramming LSP and Language Servers

Game development 4

Making Every Operation in a Web Single-Player Game Truly Atomic Making Action Sprite Frames Hit on Time in Combat From Character art to Animation Frame Sequences Decouple Idle Settlement from Call Frequency

Rust 33

Ownership and Borrowing

Ownership Model References and Borrowing Lifetimes

Type System

Primitive Types and Memory Layout Enums and Pattern Matching struct and trait

Advanced Generics and Traits

Generics and Monomorphization Trait Objects and Dynamic Dispatch Associated Types and GATs

Smart Pointers and Memory

Box, Rc, and Arc Cell, RefCell, and Mutex: Interior Mutability Allocators and Memory Layout

Error Handling

Result and Option The Error Trait and Ecosystem

Concurrency and Asynchrony

Send and Sync async/await Internals Tokio Runtime Threads and Synchronization Primitives

Unsafe Rust

Raw Pointers and FFI unsafe Blocks and the Soundness Contract Inline Assembly

Macro System

Declarative Macros: macro_rules!Procedural Macros

Cargo and Build

Cargo.toml Explained Conditional Compilation and cfg Workspace and Project Management

Standard Library Depth

Iterators and Closures Collection Types Strings and Paths

FFI and System Interaction

C ABI and bindgen Kernel Modules and no_std

Async Ecosystem

Tower and Middleware tonic and gRPC

Computer systems

Distributed systems 23

Distributed systems foundations

CAP and Consistency Models Time and Clocks Failure Models

Consensus Protocols

Raft Paxos Consensus in Practice in Real Systems

Replication and Consistency

Replication Strategies Conflict Resolution Distributed Transactions

Partitioning and Routing

Consistent Hashing Sharding Strategies Dynamic Rebalancing

Members and Discovery

Failure Detection Gossip Protocol Service Discovery

Distributed Storage

Distributed Read/Write Paths Anti-Entropy and Data Repair Storage System Instances

Messages and Streams

Message Semantics Kafka Architecture Consumer Groups and Collaboration

Observability

Distributed Tracing Metrics and Alerts

Electronics 34

Basic Electrical Engineering

Basic Physical Quantities and Laws DC Circuit Analysis

Passive Components

Resistor Capacitor Inductors and Transformers

Semiconductor

Band Theory and Doping PN Junction Diode BJT Bipolar Junction Transistor MOSFET (Metal-Oxide-Semiconductor Field-Effect Transistor)Overview of Power Devices

Analog Circuits

Fundamentals of Amplifier Circuits Operational Amplifier (Op-Amp)Power Supply Circuits

Digital Circuits

Logic Gates and Boolean Algebra Combinational Logic Circuits Sequential Logic Circuits

Interfaces and Communication

Logic Levels and Interface Physical Layers Serial Bus Protocols USB and Type-C PCIe and High-Speed Interconnects Display and Camera Interfaces

Wireless Communication

Wi-Fi and Bluetooth IoT Wireless Protocols

Power and Protection

Charging Protocols and Fast Charging Battery Technology Protection Circuits

Embedded Systems

MCU Selection and Development Ecosystem Common Peripheral Circuits Sensors Storage Technologies

Practice and Tools

Measurement Instruments and Tools PCB Design Fundamentals Debugging and Programming Interfaces

Linux kernel 49

Process Management

task_struct and the process lifecycle The Linux scheduler: from runnable tasks to CPU time Linux context switching — context_switch, switch_mm, and switch_to Signals and ptrace: From Asynchronous Notifications to Controlled Stops

Memory Management

Virtual Memory: Understanding Page Tables, Page Faults, and TLB from a Single Memory Access Instruction VMA and Maple Tree: How Address Ranges Are Established, Changed, and Accessed Concurrently Physical Memory Management: Constraints on Allocation from Continuous Blocks Slab and kmalloc: Object Reuse, Space Retention, and Lifecycle Page Reclaim and Swap: How to Free Memory Without Losing Needed Data

File System

VFS: From Pathname to Open File Page Cache and I/O: Caching, Writeback, and Persistence Filesystem Implementation: Layout, Journaling, COW, and Recovery VFS Advanced: Lock Ownership, Persistent Updates, and Data Passing ZFS: Storage Pools, Transaction Groups, and Synchronous Write Recovery NFS: Consistency, State, and Recovery of Remote Files Temporary and Pseudo Filesystems: Data, Interfaces, and Resource Boundaries

Interrupts and Clocks

Interrupt Subsystem Clocks and Timers Time Management: Time Scales, Corrections, and Reading

Synchronization Mechanisms

Locking Mechanisms and Memory Ordering RCU: Publication, Grace Periods, and Reclamation Synchronization Debugging: Dependencies, Races, and Stalls

Network Subsystem

Network Protocol Stack Overview TCP/IP Protocol Stack Netfilter and BPF/XDP Virtual Networks, Routing, and Queues RDMA: Queues, Memory Registration, and Completion Semantics

Block Device Layer

Block Device Layer NVMe and Storage

eBPF and Observability

eBPF Fundamentals eBPF Program Types Tracing and Debugging

Security Mechanisms

LSM and Access Control Kernel Hardening Crypto API and Storage Encryption

Drivers and Device Model

Device Model PCIe and USB Driver Models DRM and Graphics IOMMU and DMA API

Kernel Development and Build

Kbuild and Modules Debugging and Testing Live Patching

Containers and Primitives

Namespaces cgroup

KVM and Virtualization

KVM Architecture KVM Memory and Device Virtualization QEMU and KVM Collaboration

Power Management

cpufreq and cpuidle Suspend and Wake-up

Networking 46

L2 Data Link

Ethernet and MAC ARP and NDP Switching, Bridging, and VXLAN

L3 Network Layer

IPv4 Protocol IPv6 Protocol ICMP and PMTUD Routing and BGP

L4 Transport Layer

TCP UDP and QUIC Transport Layer Ports and NAT Traversal

DNS

DNS Protocol and Resolution DNSSEC and DNS Security DNS Privacy: DoH, DoT, DoQ, and ODoH DNS Infrastructure

TLS and PKI

TLS 1.2 Handshake TLS 1.3 Handshake Certificates and PKI mTLS and Mutual Authentication

HTTP

HTTP/1.1 HTTP/2 HTTP/3 Reverse Proxy and Load Balancing REST and API Design WebSocket and SSE

Proxy Agreement

Forward Proxy Transparent Proxy and TUN Proxy Protocol Comparison

VPN and Tunnels

WireGuard Protocol Tailscale Architecture IPsec and IKEv2 Tunnel Technology Comparison

Email Protocols

SMTP IMAP and POP3

Real-time communication

WebRTC Streaming Protocols MQTT

Messages and Events

Webhooks and Callbacks Message Queue Protocols

Security Protocols

SSH Protocol OAuth2 and OpenID Connect Authentication Framework

Network Diagnostics

tcpdump and Wireshark Network Fault Diagnosis Methodology Network Performance and Tuning

Protocol Design and Evolution

Binary Protocol Design Protocol Evolution and Compatibility

Systems programming 23

ELF and Binary Formats

ELF File Format Dynamic Linking and ld.so Loading and Startup: From ELF Entry to main

System Call ABI

Linux x86-64 System Call ABI: Registers, SYSCALL, and RCX/R11 ARM64 System Call ABI Anatomy of glibc Syscall Wrappers

Processes and Memory

Process Memory Layout Inside malloc: The glibc Allocator ASLR and Security Hardening

Signals and Exceptions

User-space Signal Handling ptrace and Debugging Interfaces

IO and File Systems

Buffered I/O (stdio)io_uring: Submission and Completion Model epoll and Event-Driven Programming

IPC and Synchronization

futex: Waiting and Waking for User-Space Locks Pipes and socketpair Shared Memory and Semaphores

Toolchain and Compilation

GCC and Clang Internals Linker Scripts and Relocation Make / CMake / Meson Build Systems

Performance and Debugging

Deep Dive into perf Valgrind and Sanitizers in Practice User-space Tracing with ftrace and bpftrace

Theory

Information theory 7

Information & probability

Entropy and Information Measures KL Divergence and Cross-Entropy

Coding & communication

Source Coding and Compression Channel Capacity and Coding

Representation & applications

Rate–Distortion and the Information Bottleneck Information Theory in LLMs Optional: Conditional Information and Data Processing

Control theory 11

Feedback & dynamics

Feedback and control Dynamic models Response and stability

Controllers & frequency response

PID control Transfer functions and poles Frequency response and stability margins

Sampling & applications

Sampling and digital control Constraints and robustness Control in computing systems

State space & optimization

Advanced: state feedback and observers Advanced: optimal control and MPC

Queueing theory 11

Arrivals & waiting

System boundaries and queueing timelines Little’s law and measurement boundaries Arrival processes and model assumptions

Queueing models

M/M/1: utilization and tail latency Variability, M/G/1 and Kingman’s approximation Multiple servers and resource pooling

Scheduling & networks

Scheduling, head-of-line blocking and fairness Finite queues and admission control Queueing networks and bottlenecks

Measurement & capacity

Measurement, load testing and discrete-event simulation Capacity planning and overload recovery

Arts & society

Economics foundations 20

Choice & exchange

Economic Choice and Analysis Specialization, Comparative Advantage, and Exchange

Prices & trade

Market Prices and Elasticity Surplus, Taxes, and Subsidies

Firms & interaction

Firm Costs, Pricing, and Competition Strategic Interaction and Cooperation Asymmetric Information and Contracts

Institutions & distribution

Externalities and Social Costs Public Goods and Common Resources Labor Markets, Wages, and Employment Income, Wealth, and Inequality

Output & prices

GDP and National Income Price Indexes and Inflation

Money & credit

Money, Banking, and Credit Interest, Saving, and Investment

Fluctuations & policy

Economic Fluctuations and Aggregate Demand Fiscal Policy Monetary Policy

Trade & growth

International Trade and Exchange Rates Productivity and Economic Growth

Music foundations 11

Rhythm, pitch & notation

Beat and Rhythm Pitch and Intervals Reading the Staff Meter and Accents Note Values and Rests

Tonality, melody & harmony

Scales and Keys Triads and Inversions Melody and Harmony

Timbre & form

Timbre and Texture Musical Form

Practice

Staff Reading Practice

Visual foundations 7

Light & volume

Value and Visual Contrast Light and Volume

Shape & space

Shape and Proportion Space and Perspective

Colour & composition

Hue, Lightness, and Chroma Color Palette Design Composition and Visual Emphasis

Tools & practice

Operations 25

Bluetooth Operations Manual Caddy Operations Manual Certificate Operations Manual Database Backup and Diagnostics Manual Disk Diagnostics Manual DNS Operations Manual Docker Operations Manual Git Operations Manual Grafana Operations Manual Helm Operations Manual Kubernetes Operations Manual Kernel Operations Manual Audio and Video Operations Manual Mihomo Operations Manual Network Device Operations Manual nftables Operations Manual Perforce Operations Manual Gentoo Package Operations Manual Prometheus Operations Manual SSH Operations Manual Storage Changes and Recovery Manual systemd Operations Manual Tailscale Operations Manual

Runbooks

wildwindstudio.com Certificate Auto-Renewal + Tencent Cloud CLB Rebinding Tencent Cloud EdgeOne + COS Deployment Guide: API Origin Fetch and Static Site

Homelab 10

What Local Large Models Can Help With: Boundaries, Scenarios, and Automation Local LLM Deployment Notes Full-Stack Monitoring: Prometheus + Grafana + Loki + Tempo + Bark Practical Home Network Architecture: Wired Mesh, Single-Arm Side Router, and Fail-Open Network Failure Recovery Manual rathole + Caddy Intranet Penetration: A Complete Solution from CGNAT to Public HTTPS Server Lifecycle: Onboarding and Removal Software Stack Inventory System Drive Backup and Recovery Scheduled Tasks and Automated Scheduling

Developer tools 6

chezmoi source state, differences and synchronisation CLI tools for finding, searching and reviewing Ghostty configuration, themes and terminal capabilities Zsh completion, previews and startup diagnosis WezTerm on Windows and across platforms Zellij sessions, layouts and editor navigation

Hyprland desktop 7

Hyprland Lua monitor coordinates, bindings and validation Quickshell and Caelestia components, services and local changes Hyprland Waybar configuration and modular themes Hyprland locking, idle policy and companion services Matugen from palette generation to application refresh Hyprland session environment and desktop integration diagnosis

Neovim 7

Neovim plugin specifications and key management Neovim editing actions, registers and project navigation Neovim language services, completion and formatting A Neovim project exercise fixing a boundary error Neovim input methods, remote clipboard and pane navigation Neovim versions, upgrades and fault isolation

Reference

PC parts LLM models Agent tools

Tags MCP connection Ask the docs Knowledge graph

中文
English
日本語
Knowledge graph
Site map

#Information theory

No matches. Try another keyword.

Information Theory: A Learning Path from Probability Distributions to CodingTheory
Optional: Conditional Information and Data ProcessingTheory
Entropy and Information MeasuresTheory
KL Divergence and Cross-EntropyTheory
Source Coding and CompressionTheory
Channel Capacity and CodingTheory
Rate–Distortion and the Information BottleneckTheory
Information Theory in LLMsTheory

© 2026 liz's doc

Privacy policy Terms of service

Link preview

If the preview is blank, open it in a new tab.

Open in new tab