Dao-AILab/flash-attention
flash-attention
Fast and memory-efficient exact attention
Builder
Dao-AILab
Dao-AILab • individual
Stars
24,477
Using upstream star count
Forks
2,910
Using upstream fork count
Open Issues
0
Activity Score
0/100
0 commits in 30d
Created
May 19, 2022
Project creation date
README Summary
FlashAttention This repository provides the official implementation of FlashAttention and FlashAttention-2 from the following papers.
Community Evaluation
Loading…AI Dev Skills
Unmapped
Algorithm OptimizationAttention Mechanism DesignAttention Mechanism OptimizationAttention MechanismsComputational Complexity AnalysisCUDA/GPU AccelerationCUDA ProgrammingGPU Kernel DevelopmentGPU Kernel ProgrammingGPU OptimizationHigh-Performance ComputingLarge Language Model TrainingLow-level Performance TuningMemory Efficiency in Deep LearningMemory-Efficient Deep LearningMixed Precision TrainingNumerical StabilityNumerical Stability in Neural NetworksTransformer ArchitectureTransformer Architecture Optimization
Tags
Algorithm OptimizationAttention Mechanism DesignAttention Mechanism OptimizationAttention MechanismsComputational Complexity AnalysisCUDA/GPU AccelerationCUDA ProgrammingGPU Kernel DevelopmentGPU Kernel ProgrammingGPU OptimizationHigh-Performance ComputingLarge Language Model TrainingLow-level Performance TuningMemory Efficiency in Deep LearningMemory-Efficient Deep LearningMixed Precision TrainingNumerical StabilityNumerical Stability in Neural NetworksTransformer ArchitectureTransformer Architecture OptimizationBenchmarkingContext EngineeringDockerEmbeddingsEvalsForkedGPU / CUDAGemmaHuggingFaceKV CacheMistralOpenAIPyTorchPythonResearch / PapersSecurityvLLM
Taxonomy
AI Trends
category
Deployment Context
Skill Areas
Transformer Architecture OptimizationAttention Mechanism DesignGPU Kernel DevelopmentCUDA ProgrammingMemory-Efficient Deep LearningComputational Complexity AnalysisHigh-Performance ComputingAlgorithm OptimizationLarge Language Model TrainingNumerical Stability in Neural NetworksTransformer ArchitectureAttention Mechanism OptimizationGPU Kernel ProgrammingCUDA/GPU AccelerationMixed Precision TrainingAttention MechanismsGPU OptimizationMemory Efficiency in Deep LearningNumerical StabilityLow-level Performance Tuning
tag
Use Cases
Efficient training of large language modelsLong-context sequence processingReducing GPU memory consumptionAccelerating transformer inferenceTraining with larger batch sizes on constrained hardwareEnabling longer sequence lengths in vision and language tasksTraining Large Language ModelsInference Optimization for TransformersLong Context ProcessingMemory-Constrained Model TrainingEfficient Sequence ProcessingAccelerating Large Language Model InferenceReducing Training Time for TransformersEfficient Fine-tuning of Foundation ModelsLong-context ProcessingMemory-constrained Deployment
Recent Activity
Updated 3 months ago
7 Days
0
30 Days
0
90 Days
0
Quality
production- Quality
- high
- Maturity
- production
Categories
RAG & RetrievalPrimaryEvals & BenchmarkingInference & ServingMLOps & InfrastructureDev Tools & AutomationLearning ResourcesSecurity & SafetyFoundation ModelsAI AgentsModel TrainingSearch & KnowledgeOther AI / ML
PM Skills
Cost & EfficiencyScale & ReliabilityData & EvaluationProduct DiscoveryAI-Native Architecture
Languages
Python100.0%
Timeline
- Project created
- May 19, 2022
- Forked
- Mar 28, 2026
- Your last push
- 3 months ago
- Upstream last push
- 2 months ago
- Tracked since
- Mar 26, 2026
Similar Repos
pgvector cosine similarity · $0
Loading…