Coding

Evaluate AI models on code generation and programming tasks

openbench provides a diverse set of coding benchmarks to assess model capabilities in code generation, problem solving, and software engineering tasks across multiple programming languages.

Available Benchmarks

HumanEval

164 hand-written programming problems testing function-level code generation capabilities.

bench eval humaneval

MBPP

Mostly Basic Programming Problems - entry-level Python programming challenges.

bench eval mbpp

SciCode

Scientific computing problems requiring domain knowledge and programming skills. (Alpha)

bench eval scicode --alpha

GMCQ

Graduate-level multiple-choice questions on computer science fundamentals.

bench eval gmcq

JSONSchemaBench

Tests ability to generate valid JSON outputs conforming to specific schemas.

bench eval jsonschemabench

Exercism

Real-world coding tasks as an agent evaluation across 5 programming languages.

bench eval exercism

Related Resources

• HumanEval Paper
• MBPP Dataset
• Exercism Platform