Tags: #llm-serving

AI Inference Framework

25.7k

sgl-project/sglang

A high-performance serving framework designed to accelerate inference for large language models and multimodal AI models.

llm-serving inference-framework multimodal-ai

LLM Inference and Serving Engine

76.3k

vllm-project/vllm

A high-throughput and memory-efficient open-source engine designed for fast, easy, and cost-effective serving of large language models.

llm inference llm serving deep learning

LLM Serving Platform

5.1k

kvcache-ai/Mooncake

A KVCache-centric disaggregated architecture for high-performance LLM serving, powering leading AI services.

llm serving kvcache disaggregated architecture

Educational Course

4.1k

skyzh/tiny-llm

An educational course for systems engineers to learn LLM inference serving on Apple Silicon by building a simplified vLLM-like system with MLX from scratch.

llm-serving mlx apple-silicon

AI Service Framework

21.9k

jina-ai/serve

A cloud-native framework for building, deploying, and scaling multimodal AI applications and services with gRPC, HTTP, and WebSockets.

ai-services multimodal-ai cloud-native