Prerequisites
System Requirements
- Python: 3.10 or higher
- Operating System: macOS, Linux, or Windows
Install UV (Recommended)
We recommend using uv for Python package management. It’s significantly faster than pip and handles virtual environments automatically.
macOS/Linux
Windows
Homebrew
curl -LsSf https://astral.sh/uv/install.sh | sh
Install openbench
Quick Install with UV
# Create a virtual environment and install openbench
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
uv pip install openbench
Alternative: Install with pip
# Create a virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install openbench
pip install openbench
Install from Source
For development or to get the latest features:
# Clone the repository
git clone https://github.com/groq/openbench.git
cd openbench
# Setup with UV (recommended)
uv venv && uv sync --dev
source .venv/bin/activate
# Or setup with pip
python -m venv .venv
source .venv/bin/activate
pip install -e .
Special Installation (Optional)
For safely running code execution benchmarks in a sandboxed environment:
Install Docker
Verify Docker
docker --version
docker run hello-world
Code generation benchmarks execute model code. We suggest using dockerized execution to ensure safety.
(e.g. humaneval --model openai/gpt-4o --sandbox docker).
