Library/vllm-turboquantForked

mitkox/vllm-turboquant

vllm-turboquant

vLLM 0.18.1rc1 with TurboQuant

View on GitHub↗Upstream mitkox/vllm-turboquant↗

Builder

mitkox

mitkox • individual

Stars

403

Using upstream star count

Forks

Using upstream fork count

Open Issues

Activity Score

0/100

0 commits in 30d

Created

Mar 25, 2026

Project creation date

README Summary

This repository contains vLLM version 0.18.1rc1 integrated with TurboQuant optimization technology. It provides enhanced performance for large language model inference through quantization techniques. The integration aims to improve inference speed and memory efficiency for LLM deployments.

AI Dev Skills

Unmapped

Quantization TechniquesModel CompressionLLM Inference OptimizationCUDA/GPU ProgrammingvLLM Framework ArchitectureTransformer Model OptimizationBatch ProcessingMemory OptimizationQuantizationModel ServingvLLM FrameworkPerformance TuningDistributed InferenceModel OptimizationLarge Language Model InferencePerformance OptimizationMemory EfficiencyTurboQuant Algorithm

Taxonomy

AI Trends

Model Quantization On-device AI Small Language Models Inference Optimization Efficient AI Model Compression Quantization-Aware Optimization

Deployment Context

Self-hosted On-premise Cloud API Edge/Mobile Cloud

Modalities

Text

Skill Areas

Quantization Techniques Model Compression LLM Inference Optimization CUDA/GPU Programming vLLM Framework Architecture Transformer Model Optimization Batch Processing Memory Optimization Quantization Model Serving vLLM Framework Performance Tuning Distributed Inference Model Optimization Large Language Model Inference Performance Optimization Memory Efficiency TurboQuant Algorithm

Use Cases

Efficient LLM Inference Cost-optimized Model Deployment Memory-constrained LLM Serving Low-latency Language Model Inference High-throughput Batch Processing Memory-constrained LLM deployment Latency-optimized inference serving Cost-reduced model serving Edge and resource-limited environment LLM inference Low-latency LLM inference Memory-constrained deployment Cost-optimized model serving Real-time language model inference

Recent Activity

Updated 17 days ago

7 Days

30 Days

90 Days

Quality

beta

Quality: medium
Maturity: beta

PM Skills

Developer Platform

Languages

Python100.0%

Timeline

Project created: Mar 25, 2026
Forked: Mar 28, 2026
Your last push: 17 days ago
Upstream last push: 7 days ago
Tracked since: Mar 27, 2026

Similar Repos

pgvector cosine similarity · $0

Loading…