A high-throughput and memory-efficient inference and serving engine for LLMs

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Making large AI models cheaper, faster and more accessible

SGLang is a high-performance serving framework for large language models and multimodal models.

Faster Whisper transcription with CTranslate2

Machine Learning Engineering Open Book

Nano vLLM

Large Language Model Text Generation Inference

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

Easily fine-tune, evaluate and deploy Gemma 4, Qwen3.5, Qwen3.6, gpt-oss, DeepSeek-R1, or any open source LLM / VLM!

💎1MB lightweight face detection model (1MB轻量级人脸检测模型)

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.

A framework for efficient model inference with omni-modality models

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

Superduper: End-to-end framework for building custom AI applications and agents.

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

An easy to use PyTorch to TensorRT converter