This repository contains code for benchmarking optimization solvers on problems from the energy planning domain, and an interactive website for analyzing the results. The live website can be viewed at:
https://openenergybenchmark.org/
All our benchmark problems are open and available as LP/MPS files that can be downloaded in one click from our website's Benchmark Problem Set page. Some problems have been generated by us using open source energy modelling frameworks, and for these we have configuration files and instructions for reproducing the problems.
For more details on how to contribute problems, see the Benchmarks README.
The benchmark runner can run solvers listed in the Solvers README. We use the last released solver version in each calendar year. 2025 solvers will be updated at the end of the year.
Understanding the project layout to help you navigate and contribute:
solver-benchmark/
├── runner/ # Benchmark execution scripts
│ ├── benchmark.py # Main entry point: Typer CLI running problems across years
│ ├── utils/ # Runner package: config, solver dispatch, run loop, etc.
│ │ └── solver.py # Individual solver runner (python -m runner.utils.solver)
│ ├── config/ # Solver registry, tuning options, eligibility rules (YAML)
│ ├── envs/ # Pixi environment manifests for each solver year
│ └── benchmarks/ # Downloaded problem files
├── benchmarks/ # Problem definitions and metadata
│ ├── pypsa/ # PyPSA-generated energy models
│ ├── jump_highs_platform/ # JuMP/HiGHS problem metadata
│ └── *_metadata.yaml # Problem definitions and details
├── website/ # Next.js website for viewing results
├── infrastructure/ # GCP VM deployment scripts (for running benchmarks at scale)
└── results/ # Output directory for benchmark results
├── benchmark_results.csv # Main results file
└── metadata.yaml # Merged metadata of all problems on the websiteThe benchmark runner currently requires Linux as it uses systemd-run to enforce memory limits on solvers, which is not available on macOS or Windows.
Supported Linux distributions:
- Ubuntu 20.04 LTS or later
- Debian 11 or later
- Other systemd-based Linux distributions
macOS / Windows users: Use the provided Docker image to run benchmarks in a Linux container. See Running with Docker below.
Ensure you have the following installed:
- Python 3.12+
- pixi
- systemd (usually pre-installed on modern Linux distributions)
The benchmark runner CLI (runner/benchmark.py, run via python -m runner.benchmark) is the main entry point for running benchmarks. It takes a list of solver configurations and a list of years as arguments, and runs the problems for each solver configuration and year. It installs pixi environments containing the solvers and other necessary prerequisites, so a virtual environment is not necessary just for running the benchmark runner. See README .
Quickstart:
- Run benchmarks
pixi run -e runner python -m runner.benchmark --solver-configurations highs-default --solver-configurations scip-default --years 2025 infrastructure/benchmarks/sample_run/standard-00.yaml- View logs and results
tail results/benchmark_results.csv # will overwrite currently committed results
tail runner/logs/*- View and analyze results by running the website locally
The CLI will save the measured runtime and memory consumption into a CSV file in results/ that the website will then read and display. Running the website locally will allow you to view and analyze results in a user friendly way. It will use the results from results/benchmark_results.csv.
You can use Docker to run benchmarks in a Linux container that has all the required dependencies.
# Build the runner image
docker build -t solver-benchmark-runner -f runner/Dockerfile .
# Run the default solvers across all years, results are written to host via a volume mount
docker run --rm \
-v $(pwd)/results:/solver-benchmark/results \
solver-benchmark-runner results/metadata.yamlThe container accepts the same flags as runner.benchmark (e.g. --solver-configurations, --years). Memory limit enforcement via systemd-run is not available inside Docker and is skipped automatically. For more details on available options, pixi env caching, and Gurobi licensing, see the runner Docker documentation.
We have cloud orchestration setup for running benchmarks on Google Cloud Platform. See Documentation.
Quickstart:
For cloud infrastructure setup, install:
gcloud auth application-default login
cd infrastructure
tofu init
tofu apply -var-file benchmarks/sample_run/run.tfvarsTo set up comprehensive benchmark campaigns, like the one available on the website, generate one with:
pixi run -e benchmarks python benchmarks/create_benchmark_campaign.py \
--campaign my-test \
--allThis selects problems from results/metadata.yaml, allocates them across VMs, and writes the OpenTofu configuration under infrastructure/benchmarks/<run-id>/ (or, with --target local, a local run script under infrastructure/local/benchmarks/<run-id>/). See Benchmark Campaign Creation for the full CLI reference (problem selection, VM/machine sizing, timeouts, solver/year selection, config files, adding your own problems or solvers), and notebooks/benchmark-runner-quickstart.ipynb for a worked example covering campaign creation, monitoring, and merging results into the website.
The website code is under website/. To run the website locally, you need Node.js (which includes npm) installed. Then, run the following commands:
cd website/
npm install
npm run build && npm run devOpen http://localhost:3000 with your browser to see the website.
To see the results from your runs, navigate to the results page.
Python environments for this repo (the runner/ CLI, notebooks/, benchmarks/
scripts, and dev tooling) are managed by pixi from the root
pixi.toml. Install pixi (see the installation instructions),
then run pixi install --all to set up all environments, or pixi install -e <env>
for just one of runner, notebooks, benchmarks, or dev. See notebooks/README.md
for notebook-specific tips (running Jupyter, downloading results from GCS).
We use the ruff code linter and formatter, and GitHub Actions runs various pre-commit checks to ensure code and files are clean.
You can install a git pre-commit that will ensure that your changes are formatted and no lint issues are detected before creating new commits:
pixi run -e dev pre-commit installIf you want to skip these pre-commit steps for a particular commit, you can run:
git commit --no-verify