v
Capabilities:chatcode
License:Apache-2.0
Open sourceAPI availableLocal runtimeOpen weightsvLLM
About
A fast and easy LLM serving engine with PagedAttention for high throughput inference. Serves open models with an OpenAI-compatible API, used widely in production deployments.
Pricing
Free
vLLM is a high-performance LLM serving engine. Its PagedAttention algorithm reduces memory waste, enabling up to 24x higher throughput than naive serving for models like Llama, Mistral and Qwen.
Key Features
- PagedAttention for memory-efficient inference.
- OpenAI-compatible server.
- Continuous batching and streaming.
- Quantization support (AWQ, GPTQ, FP8).
- Prefix caching for long prompts.
- Python API and CLI.
Pricing
vLLM is free and open source under Apache-2.0.
Getting Started
pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct
Then query it like OpenAI: curl http://localhost:8000/v1/chat/completions -d '{"model":"...","messages":[{"role":"user","content":"Hi"}]}'