Knowledge uplift
+35.8%
Benchmarks
The primary experiment is without skills vs with skills. We measure packaged knowledge, not which lab shipped the latest model.
Software Throughput
13.6×
Human intervention
5→0
Primary
Does giving Ruby-specific knowledge to an agent make it better at producing Ruby software?
Secondary
Does better knowledge produce more correct software per unit of time?
Relay
A support inbox: tickets, assignment, and a Hotwire conversation thread.
- Rails 8.1
- Hotwire
- PostgreSQL
- RSpec
Protocol
One prompt. One empty Rails app. Knowledge is the only variable — first none, then skills added in layers.
Build a small support inbox. Staff can list tickets, assign an owner, and reply in a live thread.
Full Skillfile
example/rails-conventionsexample/secure-defaultsexample/request-specsexample/hotwire-uiexample/job-queues
Without skills vs with skills
| Experiment | Tests | Time | Iterations | Tokens | Human | Throughput |
|---|---|---|---|---|---|---|
| Base model | 31/44 | 24m | 7 | 184k | 5 | 4.9 |
| + Rails conventions | 36/44 | 20m | 5 | 151k | 3 | 10.2 |
| + Testing knowledge | 41/44 | 18m | 3 | 126k | 2 | 17.3 |
| Full Skillfile | 44/44 | 15m | 2 | 103k | 0 | 66.7 |
Software Throughput
ST
∝
Correctness × Requirements met
Time × Rework
Rework is human intervention plus extra iterations. Full Skillfile on Relay is 13.6× the base model on this scale.
-
Base model
-
+ Rails conventions
-
+ Testing knowledge
-
Full Skillfile
Knowledge uplift
GPT-5 without Ruby Skills
67
GPT-5 + Ruby Skills
91
Knowledge uplift
+35.8%
| Model | Without skills | With skills | Uplift |
|---|---|---|---|
|
GPT-5 OpenAI |
67 | 91 | +35.8% |
|
Grok 4.6 Cursor |
64 | 88 | +37.5% |
|
Gemini 2.5 Pro |
58 | 79 | +36.2% |
|
Claude Opus 4.1 Anthropic |
71 | 94 | +32.4% |
|
Claude Sonnet 4 Anthropic |
63 | 84 | +33.3% |
By skill
Most effective
-
example/query-performance
-
example/secure-defaults
-
example/rails-conventions
-
example/request-specs
-
example/hotwire-ui
Highest productivity impact
-
example/request-specs
-
example/job-queues
-
example/query-performance
-
example/rails-conventions
-
example/hotwire-ui
Same Skillfile, by model
| # | Model | Score | Tests | Time |
|---|---|---|---|---|
| 1 |
Claude Opus 4.1 |
94 | 42/44 | 18m |
| 2 |
GPT-5 |
91 | 41/44 | 16m |
| 3 |
Grok 4.6 |
88 | 39/44 | 14m |
| 4 |
Claude Sonnet 4 |
84 | 38/44 | 11m |
| 5 |
Gemini 2.5 Pro |
79 | 35/44 | 22m |