November 19, 2026
Bodo Engineering Team
As Large Language Model (LLM) architectures, numerical formats, and hardware architecture continue to evolve, optimizing inferential performance has become increasingly complex. While the prefill stage processes input tokens in parallel, the decode stage generates tokens sequentially, often leading to memory-bandwidth bottlenecks and poor hardware utilization. Achieving maximum performance therefore requires both carefully optimized model computations and efficient runtime resource management.
At Bodo, our work on compilers for high-performance data processing has led us to examine how compiler techniques are being applied to LLM inference. Inference compilers typically transform model definitions into a high-level intermediate representation (IR), apply graph-level optimizations such as operator fusion and algebraic simplification, and generate or select implementations suited to the target hardware. While these systems share many underlying techniques, they differ significantly in the scope of their optimizations and how they execute model computations. Some primarily optimize individual operations or fused groups using specialized kernels, others explore alternative computation graphs to identify more efficient implementations, and newer approaches coordinate execution across entire models using persistent GPU kernels.
In this two-part series, we examine seven inference compilers and compare their compilation pipelines, intermediate representations, high-level transformations, and code generation strategies. We use common LLM operations such as matrix multiplication and attention, along with the Qwen3-0.6B architecture to illustrate how these approaches differ
Part 1: Focuses on four established, open-source compiler ecosystems: torch.compile, vLLM, XLA, and TVM.
Part 2: Explores three more specialized approaches: TensorRT/TensorRT-LLM, Luminal, and Mirage Persistent Kernel (MPK), highlighting hardware-specific optimization, search-based compilation, and persistent megakernel execution. Part 2 will be published next week.
PyTorch’s torch.compile captures and compiles parts of a PyTorch program (including but not limited to LLM inference). TorchDynamo observes Python execution and records supported operations in an FX graph. The graph is then passed through AOTAutograd and lowered by TorchInductor, which generates code for the target device or dispatches operations to optimized libraries.
AOTAutograd creates the forward graph and prepares it for compilation. It functionalizes mutations (i.e., rewrites in-place updates into operations with explicit inputs and outputs), applies operator decompositions, and makes sure every tensor has a known shape and dtype. Although not applicable to LLM inference, AOTAutograd for training will generate the backward graphs and can partition them so values needed for gradients are saved or recomputed as appropriate.
The captured and transformed graph is an FX GraphModule whose nodes commonly represent ATen operators. ATen is PyTorch’s operator library: operations such as aten.mm, aten.add, and aten.softmax define tensor computations independently of the Python syntax that produced them. TorchInductor consumes this graph, applies further lowering and fusion, and emits device code. Compilation is typically just-in-time; guards on properties such as shapes, dtypes, and devices determine when a compiled graph can be reused.
The FX graph is a directed acyclic graph (DAG). Each node represents an operation, a value, or a reference to a parameter. Tensor metadata describes shape, dtype, device, and stride. A bare matrix multiplication may appear as an ATen operation such as a call to %y = call_function[aten.mm](%x, %weight_t) although matrix mutiply more frequently occurs with a bias addition epilogue as a linear layer represented by aten.linear.
Depending on capture and decomposition, attention may appear as a higher-level aten.scaled_dot_product_attention (SDPA) operation or as smaller ATen operations, for example, QKᵀ matrix multiply, scale and mask operations, softmax, and probabilities × V matrix multiply.The graph’s representation depends on the model code and the transformations applied. An SDPA node in FX does not, by itself, specify the final attention kernel.
AOTAutograd and subsequent compiler passes transform the captured graph before code generation. AOTAutograd makes mutations explicit through functionalization, applies configured decompositions, and produces compiler-ready forward and backward graphs. Decompositions express higher-level operators in terms of simpler ATen operations. For example, an operation may become a sequence of matrix multiplication, addition, activation, and reduction nodes.
These decompositions expose the computations to later compiler passes, but not every high-level operator must be broken down immediately. Some operators (e.g., SDPA) may remain as a higher-level ATen node so that backend selection can choose a specialized implementation.
TorchInductor performs additional graph lowering, fusion, layout selection, and buffer planning. Compatible pointwise and reduction operations can be combined into generated kernels. For example, residual addition with normalization, or SiLU with elementwise multiplication, may be fused when supported by the graph and target backend. Matrix multiplications are represented as distinct operations; surrounding operations may be fused where the implementation supports it.
For SDPA, the selected backend determines how attention is implemented. An eligible fused backend can handle the attention calculation without separately materializing all intermediate score and probability tensors. If no fused backend is selected, the operation may be decomposed into matrix multiplications, scaling and masking, and softmax operations.
TorchInductor lowers the ATen graph into its internal IR, which represents tensor operations, layouts, buffers, and scheduling decisions. Matrix multiplication nodes are commonly lowered to external kernel calls, such as extern_kernels.mm, which dispatch to an optimized library implementation. Depending on the operation, device, and compiler configuration, Inductor can also generate a matrix-multiplication kernel.
Fusible pointwise and reduction operations are commonly generated as Triton kernels. The generated wrapper allocates or reuses buffers and launches those kernels, while library calls perform operations such as GEMM. For example, a generated function may call a Triton kernel for a fused normalization and an external matrix multiplication for the following projection.
Attention code generation depends on the selected SDPA implementation. A fused implementation dispatches to an attention kernel; a decomposed implementation emits separate operations for the QKᵀ product, masking and softmax, and the product with V. The final generated code therefore reflects the transformations, backend eligibility, input shapes and dtypes, and target device for that compilation.
vLLM is a high-performance LLM inference serving framework. It provides a runtime for efficient request scheduling and KV cache management, optimized re-implementations of popular open-source LLMs, and a compiler built on PyTorch's compilation infrastructure that applies additional optimizations to model computations.
These optimized model re-implementations use PyTorch as the framework, with custom operators for operations such as attention. During compilation, TorchDynamo captures the re-implemented model (if there is one, otherwise vLLM can use compatible transformers or custom model definitions) into an FX graph, which vLLM's compilation backend partitions around attention operators. The surrounding computations are compiled with TorchInductor, while attention executes through specialized runtime kernels. vLLM also integrates custom Inductor passes to perform LLM-specific optimizations, such as fusing RMSNorm with quantization. Finally, Inductor generates GPU kernels, typically using Triton, for platforms including NVIDIA CUDA, AMD ROCm, and Intel XPU.
vLLM uses the same FX graph representation produced by TorchDynamo, extended with opaque attention operators and custom vllm_ir operators. For example, a portion of the Dynamo graph for a Qwen3 decode block is shown below (simplified), before graph partitioning and compiler fusion passes run:
# ... QKV projection, RoPE, and KV cache update ...
torch.ops.vllm.unified_attention_with_output(
query_14, key_14, value_4, output_19,
synthetic_local_tmp_8_,
kv_cache_dummy_dep=kv_cache_dummy_dep_4
)
attn_output_4 = output_19.view(-1, 1536)
output_parallel_17 = torch._C._nn.linear(
attn_output_4, o_proj_weight, None
)
# ... retrieve normalization weight ...
fused_add_rms_norm_maybe_inplace_8 = (
torch.ops.vllm_ir.fused_add_rms_norm.maybe_inplace(
output_parallel_17, residual_7, norm_weight, 1e-6, None
)
)
# ... subsequent MLP operations ...
This graph contains an opaque call to their custom attention implementation, unified_attention_with_output, standard PyTorch operations that reshape and project the attention output back to the model's hidden dimension, and a custom vllm_ir.fused_add_rms_norm operator that combines residual addition with RMS normalization.
vLLM's compilation backend partitions the FX graph around attention operators, separating attention execution from the surrounding computations. The resulting graph partitions are compiled with TorchInductor, extended with custom vLLM passes that can perform additional fusions.
For example, an AddRMSNormFusionPass recognizes a residual addition followed by the custom vllm_ir.rms_norm operator and replaces them with a single vllm_ir.fused_add_rms_norm operation. In the Qwen3 example above, this fusion is already present in the model definition, so the pass does not need to apply this transformation.
During compilation, vLLM lowers its high-level IR operators to hardware-specific implementations selected from a registry based on backend priority and input compatibility. These implementations can use existing CUDA or Triton kernels, or lower to standard PyTorch operations that TorchInductor further optimizes and compiles. vLLM also performs memory optimizations using maybe_inplace annotations, which identify inputs whose storage may be reused. The compiler tracks this information through functionalization and removes unnecessary copies after lowering to potentially in-place implementations. Attention is handled separately through vLLM's attention backend system, which selects optimized implementations such as FlashAttention, FlashInfer, or Triton Attention based on hardware and model configuration.
XLA is a compiler written with a mix of MLIR and its own infrastructure. XLA typically takes a high-level dialect (e.g., StableHLO, VHLO, CHLO) and then performs the high-level optimizations that we are focused on in this blog after converting to MHLO or typically to XLA’s own internal HLO (non-MLIR) intermediate representation. After that, MLIR dialects such as THLO, GML_ST, LMHLO, tensor, and memref are used for tiling computations, scheduling, buffer assignment and explicit memory operations. Several paths can be taken at this point: 1) calls to cuBLAS or cuDNN routines, 2) use dialects such as xla_gpu, linalg, vector, arith, math, and scf then LLVM to PTX, or 3) Triton to LLVM to PTX. XLA is primarily used to produce executables for TPUs although CPUs and GPUs can also be targeted. The high-level optimizations to the graph are typically done using the HLO IR and that is the focus of the rest of this section.
XLA’s internal HLO organizes a computation into an HloModule, containing one or more HloComputation objects. One computation is designated as the entry point; others can describe, for example, the body of a reduction or a fusion.
An HloComputation is a graph of HloInstruction nodes. Each instruction has an operation code, a result shape and type, and operands that point to its inputs. Instructions may also carry attributes such as dimension numbers, layout, source metadata, sharding, or backend configuration. A computation has parameters and a root instruction that specifies its result.
Matrix multiplication is represented by an HLO dot instruction. Its attributes identify the dimensions that are contracted and, for batched matrix multiplication, the dimensions that are treated as batches. For example, one possible projection appears as:
dot.597 = bf16[5,2048]{1,0} dot(reshape.596, transpose.595),
lhs_contracting_dims={1}, rhs_contracting_dims={0}
The result has shape [5, 2048] and bf16 element type. The {1,0} notation describes its physical dimension order.
Attention in HLO can be represented either as a graph of HLO instructions or a single attention instruction. By default, attention will appear as a series of HLO instructions but XLA can do pattern matching to find a variety of attention patterns in the graph. Such patterns are replaced with single calls often to libraries such as cuDNN.
XLA applies sequences of HLO-to-HLO passes to simplify the graph and prepare it for the target. Passes may run more than once as earlier rewrites expose new opportunities.
Common passes include dead-code elimination, common-subexpression elimination, algebraic simplification, gather canonicalization, and movement or merging of reshapes. The logistic expander rewrites a logistic operation into negation, exponential, addition, and division operations. Later fusion can combine such smaller operations with their consumers.
Matrix multiplication receives additional transformations. Dot operations can be canonicalized by rearranging their batch and contracting dimensions, and eligible transposes can be folded into a dot so that a separate transpose need not be materialized. On the GPU path, passes may pad selected GEMM dimensions, rewrite eligible dots into Triton GEMM fusions or library calls, and autotune eligible implementations.
Layout assignment chooses physical layouts for tensors according to operation requirements and backend preferences. Fusion passes then group compatible operations into larger computations. In the GPU pipeline, priority fusion makes decisions using estimated profitability; other passes can create multi-output or horizontal fusions. The exact passes and their effects depend on the target and compilation configuration. It is important to note that a fusion of attention’s operators into one kernel will appear as one operator in the IR and will reduce kernel launch overheads and reduce memory traffic but custom attention operators can use implementations with different algorithms offering much larger performance benefits.
XLA also supports transformations for distributed execution. For example, an SPMD partitioner can use sharding information to partition a computation across devices and introduce communication operations such as collectives where needed.
After HLO optimization, XLA schedules operations, assigns buffers, and lowers the computation for the target. Buffer assignment determines storage for intermediate values, while copy insertion resolves cases where layout or buffer-sharing requirements cannot safely be satisfied without a copy. Scheduling and rematerialization can also influence how long values remain live and how much memory execution requires.
For GPU code generation, an HLO operation may follow different paths:
The GPU backend uses lower-level representations including Triton IR or LLVM IR, depending on the selected code-generation path, before producing device code. The executable also contains the information needed to launch kernels and invoke library calls in the scheduled order. For attention, the final executable reflects the representation and transformations selected for that particular computation.
Apache TVM is an open-source machine learning compiler stack that optimizes models at both the graph and kernel levels. It supports targets including CPUs, NVIDIA and AMD GPUs, Apple GPUs, and WebGPU, although available backends and operator support depend on the build and target.
TVM can import models from frameworks such as PyTorch and ONNX, or build them using its own Python frontend. The frontend produces an IRModule containing graph-level Relax functions. Compiler passes transform the graph, lower operators to tensor-level functions, fuse compatible computations, and schedule kernels for the target. The resulting executable combines compiled kernels and library calls with code for the Relax virtual machine, which coordinates execution.
Relax represents a model as functions containing bindings between tensor operations. Dataflow blocks group operations whose intermediate values are local to the block while designated outputs connect the block to the rest of the function. Tensor types describe properties such as shape and dtype, and dimensions can be symbolic.
For example, a linear layer with a bias can be represented as the following simplified Relax program:
@R.function
def forward(
x: R.Tensor((32, 64), "float32"),
weight: R.Tensor((128, 64), "float32"),
bias: R.Tensor((128,), "float32"),
) -> R.Tensor((32, 128), "float32"):
with R.dataflow():
weight_t = R.permute_dims(weight)
product = R.matmul(x, weight_t)
result = R.add(product, bias)
R.output(result)
return result
The weight transpose, matrix multiplication, and bias addition are separate operations connected through their results. In Relax, attention can be represented either as an R.nn.attention operator or as a graph of matrix multiplication, scaling, masking, softmax, and a second matrix multiplication. The representation depends on the imported model, and an R.nn.attention call does not by itself specify a fused attention kernel. Other functions in the same IRModule can represent lower-level tensor programs or calls to external implementations.
The CUDA pipeline illustrates how TVM connects graph and kernel optimization. LegalizeOps lowers supported Relax operations to calls to tensor-level functions. AnnotateTIROpPattern classifies those functions by their access and reduction patterns, and FuseOps uses those classifications and graph dependencies to group compatible operations. FuseTIR then combines the tensor-level functions within each group.
In the linear example, TVM can fuse the matrix multiplication and following bias addition into one group while leaving the weight transpose separate. Fusion reduces the number of kernel boundaries, but the eventual placement of intermediate values in registers or shared memory depends on the subsequent schedule.
The Qwen3 graph provides a larger example. TVM forms groups for computations including RMSNorm, residual addition, rotary position embedding, and SwiGLU activation. The following excerpt shows part of a rotary embedding group after legalization and graph fusion, but before FuseTIR combines its tensor-level functions:
lv93 = R.call_tir(Module.strided_slice5, (lv76,), ty_args=[...])
lv94 = R.call_tir(Module.strided_slice6, (lv76,), ty_args=[...])
lv95 = R.call_tir(Module.tir_negative1, (lv94,), ty_args=[...])
The group slices and rearranges parts of the input, then combines them with position-dependent values. The complete group contains fourteen tensor-level calls, which FuseTIR combines into a single tensor program.
TVM’s tensor-level IR, TIRx, represents loops, buffer accesses, reductions, and target-specific
execution details. A schedule specifies how a tensor computation is implemented without changing the computation itself, for example by tiling loops, binding work to GPU threads, and placing intermediate values in shared or local memory. DLight applies a sequence of target-specific rules and selects the first rule that can schedule a given tensor program. MetaSchedule instead explores a space of candidate schedules, builds and measures them on the target, and records the results in a tuning database. These approaches offer different tradeoffs where DLight provides a quick default schedule, while MetaSchedule spends additional compilation time searching for a better implementation.
Lowering turns the scheduled tensor programs into target code and separates host-side launch logic from device kernels. Relax’s final passes also plan storage, lower allocations and symbolic shape handling, and prepare calls to compiled or external functions. At runtime, the Relax virtual machine executes the resulting control and allocation instructions and invokes the kernels and library functions.
Across torch.compile, vLLM, XLA, and TVM, we see several common approaches to optimizing LLM inference. High-level IR transformations such as operator fusion help reduce memory traffic and kernel launch overhead, while lower-level compilation generates specialized kernels or selects optimized library implementations for operations like matrix multiplication and attention. Together, these techniques allow compilers to optimize model computations while leveraging highly tuned implementations of common operations.
In Part 2 (to be published next week), we will explore three additional approaches: TensorRT's hardware-specific optimization and tactic selection, Luminal's search over alternative computation graphs and implementations, and MPK's persistent megakernel execution model. These systems provide a different perspective on how broadly compilers can explore optimization opportunities and coordinate computation across an inference workload.



