openai/tiktoken
tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Builder
OpenAI
openai • ai-lab
Stars
18,346
Using upstream star count
Forks
1,488
Using upstream fork count
Open Issues
0
Activity Score
0/100
0 commits in 30d
Created
Dec 1, 2022
Project creation date
README Summary
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
Community Evaluation
Loading…AI Dev Skills
Unmapped
Byte Pair EncodingLanguage Model IntegrationNatural Language ProcessingText PreprocessingTokenization
Tags
Byte Pair EncodingLanguage Model IntegrationNatural Language ProcessingText PreprocessingTokenizationForkedGPTJupyterOpen SourceOpenAIPythonTransformers
Taxonomy
Deployment Context
Modalities
Skill Areas
Recent Activity
Updated 3 months ago
7 Days
0
30 Days
0
90 Days
0
Quality
production- Quality
- high
- Maturity
- production
Categories
Foundation ModelsPrimaryData Science & AnalyticsOther AI / MLLearning Resources
PM Skills
Scale & ReliabilityDeveloper Platform
Languages
Python100.0%
Timeline
- Project created
- Dec 1, 2022
- Forked
- Mar 14, 2026
- Your last push
- 3 months ago
- Upstream last push
- 19 days ago
- Tracked since
- Feb 8, 2026
Similar Repos
pgvector cosine similarity · $0
Loading…