Ecosystem & Stack: sglang

LMCache is an LLM serving engine extension designed to significantly reduce Time-To-First-Token (TTFT) and boost throughput, especially for long-context scenarios, by intelligently reusing KV caches.

A toolkit for quantizing (compressing) Large Language Models (LLMs) with hardware acceleration across various GPUs and CPUs, integrating with popular inference frameworks.

An efficient and user-friendly library for scaling Reinforcement Learning with Large Language Models on large-scale GPU resources.