paged-attention

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

A high-throughput LLM serving engine with non-uniform KV cache compression, built on vLLM

⚡ High-throughput LLM serving runtime to run frontier 70B–753B models on a single 24GB GPU (RTX 4090/L4). Features 75% KV compression, 57% subspace activation pruning, and dual OpenAI + Anthropic APIs.

Implementation of PagedAttention from vLLM paper - a breakthrough attention algorithm that treats KV cache like virtual memory. Eliminates memory fragmentation, increases batch sizes, and dramatically improves LLM serving throughput.

Rust + cuTile research prototype for paged latent-cache LLM decode attention, validated on an RTX 4060.

SILKern: sparse-index localization kernels for context-parallel decode — deterministic, allocation-free, CUDA-graph-safe

An AI inference engine from scratch. Like nanoGPT, but for serving.

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

A tested from-scratch Qwen inference engine: KV caching, continuous scheduling, paged KV, SSE, and Triton on Modal.

🌱 A tiny, readable LLM serving engine with vLLM/SGLang-style features.

Measurement harness for the sliding window attention premium in the vLLM TPU Ragged Paged Attention v3 kernel: per layer decode cost, block size control, throughput, and goodput for Gemma 4 31B on TPU v6e.

High-performance On-Device MoA (Mixture of Agents) Engine in C++. Optimized for CPU inference with RadixCache & PagedAttention. (Tiny-MoA Native)

An animated, click-through companion to IOP Systems' Inference Infrastructure blog series. Eleven modules take you from a single attention head all the way to a production serving stack — every concept rendered as a hands-on animation you can play, pause, and step through at your own pace.

A mini-vLLM: a from-scratch LLM inference server with a paged KV cache and continuous batching (GPT-2, OpenAI-compatible streaming API).

Local-first LLM stack on a single RTX 5090: QLoRA fine-tuning, exact speculative decoding, paged KV-cache, and continuous batching — served via FastAPI with a live React dashboard.

Discrete-tick simulator for KV-cache memory compaction policies in LLM inference servers. Compares NoCompaction, GreedyCompaction and ThresholdCompaction via 2D parameter sweep, Pareto frontier analysis and latency impact (P95/P99). C++20 + Python.