Repo-level coding benchmarks, SWE-bench style.

Awana runs SWE-bench-style coding benchmarks for AI labs: senior engineers write, solve and grade repo-level tasks inside your repos, harness and rubrics, and an Anti-Reward-Hacking Audit catches models gaming the grader. Awana is not a data vendor and owns no data; the work is your IP. First PR the same day an engineer gets access. Double digits inside a month.

From blind round to dedicated cell

Clock starts when your accounts go live.

Engineers who ship with coding agents every day.

They use coding agents on real work, so they know where agents cut corners. Their tasks are realistic and their grades hold up to review. Of Awana's LLM evaluation services, this is the one built for code.

01

Write and solve tasks

Repo-level tasks with a real codebase, a failing test and a reference fix. Scoped to the difficulty band you need.

02

Grade agent output

Pass or fail against your rubric with a written rationale, plus trajectory review: did the agent read the right files, run the tests, stop at the right time. For code RLHF, the cell does the same RLHF and rubric grading our embedded evaluators do, on coding tasks.

03

Build golden sets

Reference solutions, hidden tests and edge cases that make a task gradeable by a machine and defensible to a reviewer.

04

Red-team coding agents

Adversarial tasks, ambiguous specs and unsafe-change traps that expose where an agent cuts corners. Beyond code, see our AI red teaming services.

05

Calibrate the pod

Every grader is calibrated against your rubric before the first task, and we monitor agreement across the pod so labels stay consistent.

06

Work in three languages

English, Spanish and Portuguese at a professional level, so multilingual benchmarks get graded by people who read the prompt natively.

Every failed task becomes a bug report your model team can act on.

  • The full agent trajectory, so you can see where it went wrong.
  • The rubric miss, named, so the failure maps to a criterion.
  • A minimal repro, so the model team can reproduce it without reading the whole task.

Reward hacking audit: a pass has to be real.

Reward hacking is a model passing the grader without solving the task: weakened tests, hard-coded outputs, edits to the test harness. In Awana's Anti-Reward-Hacking Audit the cell checks whether the model fixed the problem or gamed the grader, then closes the gap.

  • TestsWeakened, skipped or special-cased tests that let a wrong fix pass.
  • OutputsExpected outputs hard-coded into the solution instead of computed.
  • HarnessEdits to the test harness, fixtures or grading scripts.
  • SpecUnder-specified tasks the model exploits by solving an easier problem.

What the cell hands back

  • Each gamed pass, flagged with the diff or trajectory step that shows it.
  • Hardened tasks: tighter specs and hidden tests that close the loophole.
  • Hardened graders: harness and scoring fixes, checked against your rubric.

The audit runs on your tasks, inside your environment. Findings, fixes and tests are 100% your IP.

Roughly 1 in 40 make it through.

Benchmark work punishes weak engineers twice: bad tasks and bad grades. So every person clears the same four gates before you meet them.

1 in 40Pass all four gates
  1. Live technical interview. A real conversation about real work, in the stack your benchmark targets.
  2. Role-specific take-home. About four hours. For benchmark roles it is a task-and-grade exercise worked with a coding agent: can they direct it, catch its mistakes, and know when to take over.
  3. English fluency screen. Verified live, so rubric discussions and reviews run without friction.
  4. Reference checks. We talk to people they have actually shipped with.
PythonTypeScriptGoJavaRustC++SQLDockerGitClaude CodeCursorCodex

Prove the signal first. Then scale the cell.

One operator runs your cell end to end. You talk to the person who built the team, not an account manager.

Day 0

Scope the program

A 30-minute call with Calvin Sedao, our CEO. Task type, rubric, volume, your harness and current vendors. No charge, no obligation.

Step 1 · Free

Free blind calibration round

Vetted engineers complete a small sample of your actual repo-level tasks, blind. You judge the signal before any financial commitment.

Step 2 · Week 1

Paid one-week trial pod

Two to three engineers, curated for your task type and stack, on the same tasks as your existing vendors. First PR the same day an engineer gets access. One fixed price for the week, all-in, quoted on the scope call.

Step 3

Dedicated pod retainer

A multi-engineer cell funded from your post-training or evaluation budget, with calibration and inter-rater checks running as it grows. Or walk: no obligation after the week.

What you have at the end of the week

Acceptance rate on every task, and quality and velocity next to your current vendors on the same tasks.

Rationale samples your reviewers can audit, and our recommendation on who to keep and how fast we can ramp.

Sample · invented numbers

Trial read-out: coding benchmark trial pod, week 1

  • 42tasks shipped
  • 86%accepted on first review
  • 86%agreement with your gold set, floor 80%
  • Engineer A16 tasks · 94% accepted first time · 91% agreementKeep. Strongest rationales in the pod.
  • Engineer B15 tasks · 87% accepted first time · 88% agreementKeep.
  • Engineer C11 tasks · 73% accepted first time · 76% agreementBelow your floor. We swap this seat within the week.
The format a lab receives after a one-week trial pod. Full sample read-out (PDF)

Senior, bilingual, two hours from New York.

  • Depth. A bench of engineers already calibrated on task-and-grade work, not sourced fresh for each brief. We already know who is in it.
  • Overlap. UTC-3 to UTC-6, inside your Slack and GitHub during the US workday, so grading disputes get resolved the same day.
  • Languages. English, Spanish and Portuguese, which is the edge for multilingual coding benchmarks.

For data and evaluation vendors.

Have an enterprise client on the line? A cell adds senior engineering capacity to your annotation or eval program, beside your stack, not against it.

Your clients are regulated.

Banks, fintechs, law firms, advisory firms. They don't buy a model that is pretty good at code. They buy one that doesn't ship a bug into their codebase.

Their bar is set by your graders.

A model is only as good in a client's domain as the tasks it was trained and scored on. Whoever writes and grades those tasks sets the ceiling.

Capacity behind you, not a rival.

Engineers under NDA, working behind your platform or beside it, as you set it up. We never take your client, and we own no data.

What benchmark teams ask us first.

Are these labeling contractors or software engineers?

Software engineers. Every person we place passes a live technical interview, a role-specific take-home worked with a coding agent, an English screen and reference checks before you meet them. Grading experience is a plus, engineering ability is the requirement.

What is reward hacking, and how does the audit catch it?

Reward hacking is a model passing the grader without solving the task: weakening or special-casing tests, hard-coding expected outputs, editing the test harness, or exploiting a loose spec. In the Anti-Reward-Hacking Audit the cell checks passing solutions for those patterns, then hardens the tasks and graders so the same trick stops working.

How fast, really?

A vetted shortlist in 5 business days, most pods staffed in about three weeks, and a first PR the same day an engineer gets access. The bench is already calibrated on benchmark work; on the scope call we tell you whether your stack is on it.

Can you work inside our environment and under our NDA?

Yes. An NDA and a standard DPA before any data is shared, a background check on every engineer before they touch your systems, and access through your SSO or OAuth with least privilege, removed at offboarding. Work stays in your environment.

We run an annotation or eval platform. Are you a competitor?

No. Awana is an engineering capacity partner, not a data vendor. We work behind your platform or beside it, never take your client, and do not build, broker or sell datasets. Code, rubrics and evaluation outputs stay 100% client IP.

How do you get to 10+ without quality dropping?

Same four gates for every hire, a calibration round against your rubric before the first task, inter-rater checks across the pod as it grows, and a swap within the week for anyone who falls behind. Every engineer added after the trial clears the same gates as the first three.

What are the first steps, and what do they cost?

Three steps. First, a free blind calibration round on a small sample of your actual repo-level tasks. Then a paid one-week trial pod: two to three engineers beside your current vendors, first PR the same day an engineer gets access, one fixed price quoted on the scope call. Then a dedicated pod retainer.

Can we run you beside our current vendor?

Yes, that is how the trial is built. Give both the same task set for the week and compare quality and velocity. A cell complements your stack; it does not replace the vendors doing volume work. See a sample read-out (PDF).

More questions are answered in the text version of this page.

Start with a blind round. Free, on your tasks.

A 30-minute call with Calvin Sedao, our CEO. Bring the task type, the rubric and a sample of your repo-level tasks.

Book a call