JIT Compilation

Compiles bytecode into machine code at runtime. Tiered compilation uses a baseline compiler to ensure startup speed and an optimizing JIT to boost peak performance; inline caching and speculation with deoptimization allow the JIT to perform aggressive optimizations that AOT cannot dare to attempt—if speculation fails, it falls back to interpreted execution.

Overview

Traditional compilation (AOT) generates all machine code before the program runs. JIT (Just-In-Time) compiles bytecode or IR into machine code at runtime—it only compiles "this piece of code is indeed being called." This brings two unique advantages: runtime information (knowing which branch is truly hot, which types are actually used), and no need to compile unreachable code (shrinking the compilation scope, accelerating startup). The cost is that compilation time is counted as runtime, so the core contradiction in JIT design is "compilation quality vs. compilation overhead"—this article explains how JIT resolves this contradiction using tiered compilation and speculation + deoptimization.

Tiered Compilation: Not Just One JIT, But Several Complementary Ones

Modern JITs do not have just one compiler, but are layered:

baseline/JIT (Ignition/V8, or C1/HotSpot):
  → Fast compilation, almost zero optimization
  → Each function is compiled only once, compilation time << time saved by skipping interpretation

Optimizing JIT (TurboFan/V8, or C2/HotSpot):
  → Only for "hot" functions (called many times, or loops iterating many times)
  → Performs full suite of optimizations: inlining, GVN, LICM, loop unrolling, SROA
  → Long compilation time, but hot functions account for a large portion of runtime, so it's worth it

Call counts and loop backedge counts determine "whether to upgrade tiers":

counter[function] += 1                                  ← Increment by one on each call
if counter[function] >= threshold_baseline:
    Compile as baseline code

backedge_counter[loop] += 1                             ← Increment by one on each backedge
if backedge_counter[loop] >= threshold_optimized:
    Enter optimizing compiler

Thresholds are dynamically adjusted (V8's TurboFan relaxes thresholds during the startup phase for faster cold starts; lowers them after entering steady state so hot spots are optimized faster).

Inline Caching: Fast Path for Polymorphic Calls

One of the core optimizations of JIT, targeting scenarios like "the same obj.method() call is made 1000 times, always with the same type":

This is not just done inside the compiler—inline caches are generated as a combination of data + code in machine code. Each time a call comes in, it only performs one type comparison + conditional jump (2–3 instructions), with a hit rate usually >95%. When the fast path hits, it is equivalent to treating these 2–3 instructions as a speculation: speculating that the type of this call is the same as the last one. Inline caching is a typical speculative optimization for polymorphic calls.

Speculation + Guards + Deoptimization: JIT's Unique Weapon

AOT compilers do not know runtime type distributions and can only generate conservative code. After seeing "this variable is int 999 times out of 1000," the JIT can speculate it is int, generating the optimal path + guard:

guard: if (typeof(x) != INT) goto deopt_entry      ← guard
fast_code: return x + 1                             ← Speculative path

deopt_entry:
    Save current state to stack frame (on-stack replacement)
    Switch back to interpreter or baseline code
    Redo this operation in the interpreter

The key mechanism is deoptimization: when speculation is broken, the JIT must map the state of the current optimized frame (registers, local variables) back to "the state the unoptimized IR should have at this point," then jump back to the interpreter or baseline to continue execution. This requires the compiler to maintain a mapping table from optimized code to unoptimized code (deoptimization metadata)—for each optimization point, where it is in the unoptimized IR, and which unoptimized variable each optimized variable corresponds to.

On-Stack Replacement (OSR)

A special case of deoptimization: a function is executing a loop (possibly having run 10,000 times), and optimization or deoptimization is triggered at this point. The execution state needs to be transferred from the old version to the new version in the middle of the stack frame—this is OSR. OSR requires the JIT to be able to "transpile the stack frame of the currently running code into the stack frame expected by the target code"—this relies on the completeness of deopt metadata.

Code Cache and Fragmentation

JIT-compiled code is stored in the code cache. Similar to how GC manages objects, the code cache also needs management:

• Space: Compiling too many functions causes the code cache to fill up → discard less hot compiled code (Code Cache Flush), falling back to interpreted execution or baseline.
• Fragmentation: Compiled functions have inconsistent sizes, and frequent flushes cause fragmentation → the code cache uses a slab allocator (similar to malloc's arena), pre-splitting for common sizes.

The Boundary Between Tiered and AOT is Blurring

Many modern runtimes are no longer pure JIT:

• Android ART: AOT compiles dex bytecode into machine code at install time (utilizing install time), and only supplements with JIT for hot spots at runtime.
• GraalVM Native Image: AOT compiles everything to a native binary — no JIT, but uses speculative optimizations (e.g., devirtualization if class hierarchies are known to be closed).
• Wasm JIT: Running Wasm in browsers, first doing fast baseline compilation (a few milliseconds), then optimizing JIT for hot spots (tens of milliseconds)—this is the same engine as JS JIT.

The commonality between JIT and AOT is: a compiler = translating from one representation to one closer to machine representation; the difference lies only in "when the translation occurs." Tiered compilation splits this time into multiple layers, ensuring each translation happens at the most suitable moment.

Trade-offs and Failure Modes

• Compilation time eats up benefits: Short-lived scripts run only once, compilation is slower than interpretation → use tiered compilation, only heavily recompile hot spots, prefer interpretation first.
• Over-aggressive speculation: Type speculation continuously misses on bizarre inputs → deoptimization storm (continuous deopt → recompile → deopt) — monitor deopt rate, stop re-optimizing the function if it exceeds a threshold.
• Code cache full: Continuously compiling new code in long-running processes → set a code cache upper limit, use LRU eviction, degrade to baseline/interpretation when full.
• Debugging hell: The mapping between JIT machine code and source code is dynamic — deopt changes stack frames, inlining changes call stacks → debugging information must map "optimized call stacks" back to "source-level call stacks" (JVM's -XX:+PrintDeoptimizationDetails, V8's --trace-deopt).

References

• V8: Ignition (interpreter) + Sparkplug (baseline JIT) + Maglev/TurboFan (optimizing JIT) — V8 blog has design documents for each layer
• JVM: HotSpot C1 (client compiler) + C2 (server compiler), observe tiering with -XX:+PrintCompilation
• Aycock (2003): "A Brief History of Just-In-Time" — An academic survey of early JITs

Keywords: JIT, just-in-time, method JIT, tracing JIT, tiered compilation, baseline compiler, optimizing compiler, hotspot, inline caching, hidden class, speculation, guard, deoptimization, on-stack replacement, OSR, deopt metadata, code cache, PGO, profile-guided optimization, AOT