SimReal · Xitadel-QuantBench · public preview
Can an AI agent out-trade the best human strategy?
An agent receives historical order books, product rules and position limits. It researches, writes a Python strategy and ships it. We replay that strategy on a market day it has never seen and score it against the best human competition strategy on that same day.
No model has cleared the human bar overall.
Overall score
Average over 7 tasks · human reference = 80 · twice the combined human performance = 100
| # | Model | Score / 100 | Tasks ≥ human |
|---|---|---|---|
| 1 | GPT 6 | 5 / 7 | |
| 2 | GLM 5.3 | 1 / 7 | |
| 3 | Kimi K3 | 0 / 7 | |
| 4 | DeepSeek V4 Pro | 0 / 7 |
Human reference (80)
Per-task scores
Darker is higher · outlined cells reach or beat the human reference on that task
| T01Single-product market making | T04Baskets & constituents | T05Underlying & call options | T06External conversion | T09Single-product (suite 2) | T11Underlying, options & spot | T13Multi-asset, disclosed identities | |
|---|---|---|---|---|---|---|---|
| GPT 6 | 80.0 | 89.5 | 78.4 | 81.0 | 80.7 | 80.0 | 51.3 |
| GLM 5.3 | 35.5 | 96.0 | 16.4 | 15.5 | 22.0 | 4.7 | 20.8 |
| Kimi K3 | 65.8 | 0.0 | 6.7 | 45.3 | 61.1 | 14.9 | 0.0 |
| DeepSeek V4 Pro | 41.5 | 0.0 | 9.4 | 59.4 | 58.9 | 0.0 | 2.9 |
At or above human reference
- Allocation. Published results use a 2h/4h pilot allocation; the standard protocol is 6h/12h. The allocation is identical for every model in a comparison.
- Verifier. Strategies execute against a historical order book and are scored on realized P&L, maximum drawdown and Sharpe. No judge model; the same code yields the same number every time.
- Human reference. Replayed from published competition submissions under the same isolation, test episode and metric convention as every model run.
- Held-out day. The scored episode is never published;
COMMITMENT.jsonin the repository publishes its hash so the evaluation can be audited. - Xitadel is SimReal's trading environment. The name implies no affiliation with any financial institution. · simreal.co