JIT Compilation

JIT (Just-In-Time) compilation transforms a certain representation into executable code during program execution. The input can be bytecode, IR, or other representations; the compilation unit can be a function, loop, trace, or module. Whether it is triggered on the first call or pre-compiled in the background is determined by runtime policies.

The key trade-off is whether the compilation cost can be amortized by subsequent execution benefits. It requires generating fast code while avoiding overly expensive optimizations for short-lived code.

A Simple Profit Model

Let the cost of optimized compilation be $C$, the execution time of the old code be $t_0$, the new code be $t_1$, and the expected number of remaining executions be $K$. Ignoring other costs, time is only net-saved if K(t₀−t₁)>C.

For example, if compilation takes 4ms and saves 2μs per execution, it needs to be executed at least about 2000 times to offset the compilation cost. In reality, one must also consider profiling, guards, code space, caching, and deoptimization costs. This explains why "generated code is faster" does not necessarily make an entire short program faster.

The Break-Even Point Applies Only to the Remaining Workload

A 4ms compilation cost and a 2μs saving per call only break even at 2000 executions; to gain a net benefit, the number of executions must exceed this threshold. How many times the hotspot was executed previously does not directly enter this future profit account, because the interpretation time already spent cannot be recovered through this compilation.

compile_us = 4000
saved_per_call_us = 2
def gain(remaining_calls):
    return remaining_calls * saved_per_call_us - compile_us
assert gain(1000) == -2000
assert gain(2000) == 0
assert gain(3000) == 2000
# Assume an additional deoptimization/recompilation cost of 600 μs.
assert gain(3000) - 600 == 1400

Real-world decisions face uncertain remaining execution counts and failure rates. Optimizing a loop that is about to end may be counterproductive; background compilation also consumes CPU and memory bandwidth. You cannot remove compilation costs from the resource ledger of the entire service just because the compilation thread is different from the request thread. The above is merely a serial time budget model, not a direct formula for wall-clock gains during parallel compilation.

Can past hot execution repay a new compilation?

Compare two strategies from now onward. More past execution adds the same sunk cost to both, leaving future net savings unchanged. Past cost stays separate, while compilation competes only with the remaining K runs. Equality breaks even rather than profits; real systems must predict K and can be wrong.

$$\mathrm{gain}=K(t_0-t_1)-C$$

Tiering: Reduce Startup Cost First, Then Invest in Hotspots

Not all runtimes have all tiers, nor is it guaranteed that every function passes through each tier. Strategies can refer to call counts, loop back-edges, code size, existing feedback, and system resources; invalidation or environmental changes may also trigger recompilation.

Taking V8's public design as an example, Ignition is the interpreter, Sparkplug is the baseline compiler that generates machine code from bytecode; Maglev's design demonstrates how an intermediate optimization layer balances compilation speed and code quality. These are used to understand roles, not to treat the tier configuration in a specific historical article as a fixed pipeline for all versions. Sparkplug, Maglev

Feedback and Inline Caching: Remembering What Happened at This Access Point

Suppose the same source code location repeatedly executes obj.x. Generic property lookup may need to interpret object layout, property descriptors, and inheritance relationships. Inline caching records conditions observed at this access point, such as the shape, along with corresponding fast operations.

Monomorphic caches retain one shape, polymorphic caches retain a few shapes; after observing many different shapes, one can switch to a lookup strategy suitable for high polymorphism. Different systems use different structures and thresholds.

Caches are usually associated with the access point and its context, not treating "the type of the last arbitrary object" as a global answer. When object layout, prototype chain, or method definitions change, guards or dependency invalidation mechanisms are needed to prevent the cache from still pointing to outdated implementations. ICs can also exist in the interpreter; it is not only optimized machine code that can use them.

Speculative Optimization: Assumptions Must Be Explicitly Protected

JIT observes that variables are mostly integers and can generate an integer fast path, but the language may allow floats, strings, or big integers to appear next. Guards verify relevant preconditions; on mismatch, execution enters a generic slow path or reverts to a lower optimization tier.

AOT can also use speculation through static analysis, PGO, code versioning, and runtime checks. The difference lies in the timing of feedback acquisition and code generation, not that AOT never knows runtime information while only JIT can optimize.

Deoptimization: Restoring State at the Language Level

Optimization places variables in registers, removes redundant values, inlines functions, and even splits an object into several scalars. When speculation fails, the runtime must reconstruct the logical state needed for subsequent execution: current bytecode position, local variables, operand stack, inlined call frames, and necessary objects.

Therefore, deoptimization metadata records how to restore a lower-level representation from the current machine state. Some values can be read directly, while others need to be rematerialized according to the records. Saving just a few registers is insufficient to restore a program that has undergone inlining and scalar replacement.

Recovery points must also consider side effects. Suppose a deduction has already been executed, and then a guard fails; you cannot restart from the beginning of the function, as that would cause the deduction to happen again. Deoptimization must continue to a semantically correct position; it is not accurate to simply say "go back to the interpreter to redo all operations."

A Deoptimization State Model with Side Effects

Assume the teaching language's integers can grow to arbitrary precision, while the optimized version only handles the int32 range. The following Python code simulates "checking preconditions before operation, and restoring i and total upon failure"; it does not generate machine code, nor does it use Python interpreter's own JIT behavior as evidence.

MIN32, MAX32 = -(1 << 31), (1 << 31) - 1

def resume_generic(values, i, total):
    while i < len(values):
        total += values[i]
        i += 1
    return total

def specialized(values, effects):
    effects.append("begin")
    total = 0
    i = 0
    while i < len(values):
        item = values[i]
        candidate = total + item  # Model uses arbitrary precision to calculate guard condition first
        if not (MIN32 <= item <= MAX32 and MIN32 <= candidate <= MAX32):
            snapshot = {"pc": "loop_add", "i": i, "total": total}
            return resume_generic(values, i, total), snapshot
        total = candidate
        i += 1
    return total, None

effects = []
result, snapshot = specialized([MAX32, 1, 2], effects)
assert result == MAX32 + 3
assert snapshot == {"pc": "loop_add", "i": 1, "total": MAX32}
assert effects == ["begin"]
result, snapshot = specialized([10, 20], [])
assert result == 30 and snapshot is None

When the guard fails, i=1 indicates that values[1] has not yet been added to total. If restored to i=2, the value 1 would be missed; if restored to the beginning of the function, begin would be executed again. Real-world generators might use machine addition with overflow flags followed by branching, but deoptimization metadata must be able to restore the correct original operands and recovery position, and cannot pass off already wrapped machine results as language results.

If another function was inlined here, there might be only one machine stack frame on the low-level stack, but logically two call frames need to be restored. If an object has already been scalar replaced, it needs to be rematerialized while maintaining the required identity relationships; "generating a new object with identical fields" does not always maintain reference equality semantics. In V8 Maglev's design, frame state recording is a key part connecting speculative optimization with correct recovery. Maglev Deoptimization

Where can deoptimization resume without repeating effects?

A resume state must identify the unexecuted operation, computed values and completed effects. Two intentionally wrong strategies remain comparable: restarting may preserve the sum while repeating begin; skipping may preserve effect count while losing a term. Correct continuation requires checking value, position and effects together.

OSR: Switching Execution Versions Before Function Return

A loop that is called only once but runs for a long time is too late to use optimized code by the time the next function call happens. On-Stack Replacement (OSR) transfers the currently active state to another version of the code at a suitable point during execution.

Common use cases include entering optimized code from an interpreted or baseline hot loop; some runtimes also use the broader term stack replacement to describe downgrade transformations. It shares the state transition problem with deoptimization, but cannot be simply defined as "a special case of deoptimization": upgrading OSR is precisely a transfer to a more optimized version.

OSR Entry Points Must Specify the Source of Each Active Value

For example, if the loop has already processed values[0:3] and is about to process index 3, the new code must receive the current array reference, i=3, and the accumulated total; it cannot use the standard function entry initialization of i=0 and total=0. Temporary values no longer used outside the loop do not need to be transferred, but values needed later must have a definite source.

OSR compilation may also find that assumptions no longer apply during generation, requiring verification of corresponding conditions upon entry. Heat counting is responsible for deciding if it is worth attempting, while entry guards confirm that it is safe to switch at this moment; the two cannot replace each other.

Code Cache: Safe Publication and Recycling After Generation

JIT needs to allocate code space, handle relocations, register GC and exception unwinding information, and publish executable code and synchronize instruction caches according to platform requirements. It does not need to write each function as a .so file first and then dlopen; runtimes can usually directly manage code and symbols in memory.

Before recycling code, it must be confirmed that no threads are still executing it, and return addresses, function entries, or cache references cannot be dangling. What allocator, eviction strategy, and write-execute permission conversion to use is determined by the implementation and platform constraints; there is no unified rule that all code caches use slab + LRU.

Evaluate by Separating Several Stages

Do not just measure a short loop after warm-up and generalize the results to the overall performance of the service. AOT and JIT can also be combined, reducing runtime work through caching, offline profiles, or pre-compiled base code.

Related chapters: Garbage Collection, Linking and Loading.

References

• V8 Sparkplug Design.
• V8 Maglev Design.

Keywords: JIT, tiered compilation, inline cache, speculation, guard, deoptimization, OSR, code cache, AOT, PGO