LLM evaluation services, run by embedded senior engineers.
Awana offers LLM evaluation services for AI labs and enterprise AI teams: dedicated cells of calibrated senior software engineers who work inside your own evaluation harness, Slack, GitHub and rubrics, in UTC-3 to UTC-6. It is an engineering capacity partner, not a data vendor or a platform: Awana does not build, broker or sell datasets, and every output is your IP.
We took a frontier AI lab to 50 engineers on AI-training and evaluation work in three months.
A cell, inside your environment
What a cell does
- Repo-level coding tasks: write, solve, grade
- Anti-reward-hacking audits of passing solutions
- Multi-step agent trajectory evaluation
- OWASP-mapped red teaming and production stress-testing
- Rubric grading and preference data, with written rationales
- Day-zero onboarding of open-weight releases
An engineering capacity partner, not a data vendor.
Most labs already run several kinds of supplier. As an AI evaluation partner, Awana adds the senior-engineering layer beside them, in the same harness.
| Supplier type | The job it does | How a cell sits beside it |
|---|---|---|
| Annotation and labeling platforms | Volume: large pools of annotators on well-specified tasks | The cell takes the code and agent tasks in the same platform. The platform keeps its work. |
| Student and graduate networks | Breadth: many subjects and languages covered at volume | The cell covers software and agentic engineering, under the same rubrics. |
| Expert networks | Depth: credentialed specialists by the hour in medicine, law, finance, science | The cell brings production engineering judgment to the same evaluation program. |
| Awana evaluation cell | Calibrated senior software engineers, dedicated to your project, in US hours | Plugs into your harness and vendors. Builds, brokers and sells no datasets. |
Code, rubrics and evaluation outputs stay 100% your IP. Public vendors, from their own material: Mercor alternatives.
Prove the signal before you commit.
Three steps: a free blind calibration round, a paid one-week trial pod, then a dedicated pod retainer. The cell is the standing team on your work; its first week runs as the trial pod.
- Step 1 · Free
Blind calibration round
Vetted engineers work a small sample of your real repo-level tasks or red-team prompts, blind. You judge the signal before you pay.
- Step 2 · One week, paid
Trial pod
Deployed beside your existing vendors on the same tasks, so you benchmark quality and velocity on your own work.
- End of the week
Read-out
Acceptance rate on every task, rationale samples, and agreement across the pod. No obligation after the week.
- Step 3 · Ongoing
Dedicated pod retainer
A multi-engineer cell on your project, funded from your post-training or evaluation budget.
Sample · invented numbers
Trial read-out: coding-benchmark pod, week 1
- 42tasks shipped
- 86%accepted on first review
- 86%agreement with your gold set, floor 80%
- Engineer A16 tasks · 94% accepted first time · 91% agreementKeep. Strongest rationales in the pod.
- Engineer B15 tasks · 87% accepted first time · 88% agreementKeep.
- Engineer C11 tasks · 73% accepted first time · 76% agreementBelow your floor. We swap this seat within the week.
Four LLM evaluation services. Pick yours.
Each cell runs one kind of work: human evaluation for LLMs, SWE-bench-style coding benchmarks, adversarial testing, or onboarding new open-weight releases.
Embedded evaluators
Calibrated engineering evaluators inside your pipeline: rubric grading, preference data and agent evaluation, plugged into your harness.
Inside your harness · no dataset ownership Embedded evaluatorsCoding benchmark cells
Senior engineers who write, solve and grade repo-level coding tasks, with audits that keep models from gaming the benchmark.
Write, solve, grade · anti-reward-hacking audits Coding benchmark cellsAdversarial evaluation cells
OWASP-mapped red teaming, multi-step agent trajectory evaluation and production stress-testing, scoped to software and agentic engineering.
OWASP Top 10 for LLMs · no CBRN testing Adversarial evaluation cellsModel onboarding cells
LLM and evaluation engineers who bring up new open-weight releases on your stack, run conformance tests and reproduce published benchmarks.
A new offer · a pilot on one release first Model onboarding cellsHiring individuals onto your own team instead: hire AI trainers and evaluators. Evaluating memory in a production agent: AI agent memory.
How an engineer qualifies for your cell.
- Step 1
Four vetting gates
A live technical interview, a take-home built for the role, an English screen and reference checks.
- Step 2
Blind calibration
Graded against your gold set. Passing is required to join the cell.
- Step 3
Onboarding
Your guidelines, your rubrics, your Slack, GitHub and harness, and your access rules.
- Every batch
Live cell
Agreement tracked against the floor you set, with second-pass review on a sample.
Your environment, your rules.
You control
- Your Slack, GitHub and evaluation harness
- Your SSO or OAuth, least privilege, logged sessions
- Your access tiers and redaction rules, with PII masked before human review
- Access removed at offboarding
- Your code, rubrics and evaluation outputs: 100% your IP, in your environment
Awana guarantees
- NDA and a standard DPA before any data is shared
- A background check on every engineer before they touch your systems
- United States-led, working UTC-3 to UTC-6
- One accountable lead for each cell
- No datasets of our own: Awana does not build, broker or sell them
The full list for a security or procurement review: Security and contracting.
A frontier AI lab: 50 engineers on AI-training and evaluation work in three months.
We do not publish the lab's name.
- 5 business days to a vetted shortlist
- About three weeks to staff most pods, calibration included
- The same day an engineer gets access, a first PR
- Roughly 1 in 40 engineers make it through our vetting
- UTC-3 to UTC-6: full overlap with the US workday
Who you talk to.
Calvin Sedao
CEO and Co-Founder, Seattle
Founded Awana in 2020 after building a point-of-sale company with a Venezuelan engineering team.
Javier Dutan
Co-Founder and COO, New York
Former White House US Digital Service.
Your first call is with Calvin Sedao, our CEO, and one lead runs your cell end to end. Awana's leadership holds an active US security clearance.
Questions a lab asks first.
What is an engineering evaluation cell?
An engineering evaluation cell is a dedicated team of calibrated senior software engineers, with one accountable lead, that works inside your Slack, GitHub and evaluation harness, under your rubrics and access rules, in UTC-3 to UTC-6. Awana supplies the engineering capacity. The tools, data and outputs stay yours.
We already run an annotation platform and a harness. Does a cell replace them?
No. A cell is built to complement your stack. Annotation platforms, crowd vendors, student networks and expert networks cover volume and breadth. A cell adds senior engineering capacity for code and agent work, inside the same harness and beside the same vendors.
How are engineers calibrated?
Each passes the four vetting gates, then a blind calibration round against your gold set before touching live work. When your rubric changes, the cell re-calibrates before the next batch.
What is an anti-reward-hacking audit?
Engineers check whether a model's passing solutions really solve the task or game the grader: weakened or special-cased tests, hard-coded expected outputs, edits to the test harness, or exploits of an under-specified task. Then they harden the tasks and graders.
How do you handle confidential models and data?
Awana signs an NDA and a standard DPA before any data is shared, background-checks every engineer before they touch your systems, and keeps all work inside your environment, under your SSO, access tiers and redaction rules. Security and contracting.
How does an engagement start, and how is it priced?
With a free blind calibration round. Next is a paid one-week trial pod beside your existing vendors, with no obligation after the week. Then a dedicated pod retainer, funded from your post-training or evaluation budget. You get a quote after the first call.
Does Awana sell datasets?
No. Awana is an engineering capacity partner, not a data vendor: it does not build, broker or sell datasets. Every piece of code, every rubric and every evaluation output a cell produces is 100% your IP and stays in your environment.
What do you not do?
Awana does not run bio, chemical, radiological or nuclear (CBRN) risk testing. Adversarial evaluation cells are scoped to software and agentic engineering. Dangerous-capability evaluation of frontier models is led by specialist evaluators, and Awana supplies trained capacity to them when they need it.
More questions are answered in the text version of this page.
Security and contracting · Mercor alternatives · Also from Awana: Staff augmentation · Hire AI engineers · AI development · Private equity
Scope your first cell.
Thirty minutes on the tasks, the rubric and your harness. You leave with a cell design and a calibration plan. The blind calibration round is free.
We do not build, broker or sell datasets. Code, rubrics and evaluation outputs stay yours.