> ## Documentation Index
> Fetch the complete documentation index at: https://openbench.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Cybersecurity

> Evaluate AI models on cybersecurity tasks

> ⚠️ **Manual install required:** the `openbench-cyber` plugin packages real exploit code and forensic artifacts that many anti-malware tools flag. Review the risks, then install intentionally with `uv pip install "openbench-cyber @ git+https://github.com/groq/openbench-cyber.git@d93522ba70392cdceddb83f762c78a68923e70da"`.

The cybersecurity benchmarks live in the **openbench-cyber** plugin. Because of the sensitive payloads we do not ship it as a transitive dependency or optional extra—installing the plugin is an explicit opt-in. Once installed, these tasks automatically appear in `bench list`.

The plugin currently ships the **CTI-Bench** suite for evaluating model performance on cyber threat intelligence tasks. CTI-Bench provides four benchmarks covering threat extraction, vulnerability classification, security knowledge, and risk assessment.

## Available Benchmarks

<CardGroup cols={2}>
  <Card title="ATE: ATT&CK Technique Extraction" icon="crosshairs">
    Extracts MITRE ATT\&CK techniques from malware and threat descriptions.

    ```bash theme={null}
    bench eval cti_bench_ate 
    ```
  </Card>

  <Card title="MCQ: Security Knowledge" icon="brain">
    Multiple-choice questions on CTI standards, threats, detection strategies, and best practices from NIST and MITRE.

    ```bash theme={null}
    bench eval cti_bench_mcq 
    ```
  </Card>

  <Card title="RCM: Vulnerability Classification" icon="bug">
    Maps CVE descriptions to CWE (Common Weakness Enumeration) categories.

    ```bash theme={null}
    bench eval cti_bench_rcm 
    ```
  </Card>

  <Card title="VSP: CVSS Score Prediction" icon="gauge">
    Calculates CVSS scores from vulnerability descriptions.

    ```bash theme={null}
    bench eval cti_bench_vsp
    ```
  </Card>
</CardGroup>

### CyBench (CTF Challenges)

`cybench` delivers end-to-end agentic CTF scenarios powered by [`inspect-cyber`](https://pypi.org/project/inspect-cyber/). Because the default sandbox grants elevated network access, you must acknowledge the risks before running:

```bash theme={null}
export CYBENCH_ACKNOWLEDGE_RISKS=1
bench eval cybench
```

<Tip>
  Pass `--sandbox k8s` to use a Kubernetes sandbox when the optional `inspect_k8s_sandbox` dependency is available.
</Tip>

<br />

## Related Resources

* [MITRE ATT\&CK Framework](https://attack.mitre.org/)
* [NIST Cybersecurity Framework](https://www.nist.gov/cyberframework)
* [CVE Database](https://cve.mitre.org/)
* [CWE Database](https://cwe.mitre.org/)
* [CVSS Specification](https://www.first.org/cvss/)