Library/human-evalForked

openai/human-eval

human-eval

Code for the paper "Evaluating Large Language Models Trained on Code"

View on GitHub↗Upstream openai/human-eval↗

Builder

OpenAI

openai • ai-lab

Stars

3,185

Using upstream star count

Forks

442

Using upstream fork count

Open Issues

Activity Score

0/100

0 commits in 30d

Created

Jul 6, 2021

Project creation date

README Summary

HumanEval is a dataset and evaluation framework for measuring the code generation capabilities of large language models. It consists of 164 hand-written programming problems with unit tests, designed to evaluate whether models can generate functionally correct Python code from natural language descriptions. The repository provides tools for running evaluations and measuring pass@k metrics for code completion tasks.

AI Dev Skills

Unmapped

Large Language Model EvaluationCode Generation AssessmentNatural Language to Code TranslationAutomated Code TestingMachine Learning BenchmarkingProgramming Language Understanding

Taxonomy

AI Trends

Code Generation Models AI-Assisted Programming Language Model Evaluation

Deployment Context

Self-hosted

Industries

Developer Tools

Modalities

Code Text

Skill Areas

Large Language Model Evaluation Code Generation Assessment Natural Language to Code Translation Automated Code Testing Machine Learning Benchmarking Programming Language Understanding

Use Cases

Code Generation Evaluation Language Model Benchmarking AI Coding Assistant Assessment Model Performance Comparison

Recent Activity

Updated 1 years ago

7 Days

30 Days

90 Days

Quality

research

Quality: high
Maturity: research

PM Skills

Developer Platform

Languages

Python100.0%

Timeline

Project created: Jul 6, 2021
Forked: Mar 14, 2026
Your last push: 1 years ago
Upstream last push: 1 years ago
Tracked since: Jan 17, 2025

Similar Repos

pgvector cosine similarity · $0

Loading…