Engineers who train and evaluate AI models.
Working senior software engineers, in a dedicated pod, for human data, coding benchmarks and red-teaming. They work inside your tools, under your access rules, in US time zones.
We took a frontier AI lab to 50 engineers on AI-training work in three months.
What a pod does
- Preference and RLHF data, with written rationales
- Rubric grading against your rubric, blind
- Agent and computer-use trajectory review
- Repo-level benchmark tasks: write, solve, grade
- Red-team sprints under your safety policy
- Golden sets, in English, Spanish and Portuguese
Where a pod of working engineers fits.
Labs buy human data from four kinds of supplier. Each fits a different job, and many labs use more than one.
| Supplier | Who does the work | Fits when |
|---|---|---|
| Crowd and labeling vendors | Large pools of general annotators | The task is high volume and the instructions are simple |
| Student and graduate networks | University students and recent graduates, trained on task types | You need many subjects and languages covered at volume |
| Expert networks | Credentialed specialists, engaged by the hour | The question is deep domain knowledge: medicine, law, finance, science |
| Awana | Working senior software engineers, in a dedicated pod | The work is code and agents and needs production judgment: repo-level tasks, trajectory review, red-teaming coding agents |
Named vendors, side by side: Mercor alternatives for human data and AI evaluation.
Compare us on your own tasks.
Run a pod beside your current vendor on the same tasks for a week. The calibration round before it is free.
- Free
Calibration round
A blind test against your gold set. You see how closely we agree with your reference answers before you pay for anything.
- Week 1
Paid trial
Two to three engineers on your task type, inside your platform. One fixed price for the week.
- End of the week
Read-out
Acceptance rate on every task, rationale samples, and agreement across the pod.
- Your call
Keep or walk
No obligation after the week. Keep the engineers you want, or walk.
What the read-out looks like: Awana for AI labs, with a sample read-out (PDF).
Three kinds of work. Pick yours.
Human data for AI labs
Preference data, rubric grading and agent evaluation from engineers who understand the outputs they grade.
Paid pilot first · then hourly per evaluator Human dataCoding benchmark pods
Engineers who write, solve and grade the repo-level tasks coding models are measured against.
One-week paid trial · first PR within hours Benchmark podsRed teaming and evaluation
Independent testing of models and agents, mapped to the OWASP Top 10 for LLMs, with a report your risk team can file.
Fixed-price project · retest included Red teamingHiring evaluators onto your own team: Hire AI evaluators. Building memory for a production agent: AI Agent Memory. Hosting open models on your own cloud: Model onboarding pods.
How an engineer qualifies for your pod.
- Step 1
Domain screen
Engineers matched to your task, then a skills test on a real task in your domain.
- Step 2
Blind calibration
Graded against your gold set. Passing is required to join the pod.
- Step 3
Onboarding
Your guidelines, your tools and your access rules.
- Every batch
Live pod
Agreement tracked against the floor you set, with second-pass review on a sample.
Your environment, your rules.
You control
- Your platform, tools and virtual desktops
- Your SSO, least privilege, logged sessions
- Your access tiers and redaction rules
- Access removed the day someone rolls off
- Your tasks, rubrics and data, which stay in your environment
Awana guarantees
- NDA and a standard DPA before any access
- A background check on every evaluator
- United States-led, in US time zones
- One accountable lead for each pod
- No datasets of our own: Awana does not build or sell training data
The full list for a security or procurement review: Security and contracting.
Free calibration. Then paid work you can judge.
Every line starts small: a pilot, a one-week trial or a single sprint. You get a quote after the first call.
- Calibration round
- Free. A blind test against your gold set, before any paid work.
- Pilot pod
- A small calibrated pod on one task. A short, paid, fixed-price project.
- Dedicated pod
- The same evaluators on your project, month after month. Hourly, per evaluator.
- Surge capacity
- Extra calibrated evaluators for a launch or an eval push. Hourly, up or down any month.
- Benchmark trial
- One week, two to three engineers, one fixed price. No obligation after the week.
- Evaluation sprint
- One model or agent, a golden set, blind grading and a report. A fixed-price project.
- Red-team engagement
- Attack sets, severity-rated findings and a retest. A fixed-price project.
A frontier AI lab: 50 engineers on AI-training work in three months.
We do not publish the lab's name.
- Days to the first evaluators on a typical pod
- About three weeks to staff most pods, calibration included
- Within hours of accounts going live, the first PR on a benchmark trial
- Roughly 1 in 40 engineers make it through our vetting
- Two hours or less between every hub and US Eastern
- About half of US rates for the same seniority
Who you talk to.
Calvin Sedao
CEO and Co-Founder, Seattle
Founded Awana in 2020 after building a point-of-sale company with a Venezuelan engineering team.
Javier Dutan
Co-Founder and COO, New York
Former White House US Digital Service.
One lead runs your pod end to end. Awana is United States-led, and its leadership holds an active US security clearance. Calls book on Calvin's calendar.
Questions a lab asks first.
Who does the work?
Software engineers who grade AI output for clients today, drawn from the same vetted pool as our 500+ placements. Awana is United States-led, with an engineering team in Latin America that works in US time zones.
How are evaluators calibrated?
Each passes a domain screen, a skills test on a real task and a blind calibration test against your gold set before touching live data. When your rubric changes, the pod re-calibrates before the next batch.
How do you measure quality?
We track inter-rater agreement on every batch against a floor you set, run second-pass review on a sample, and log ambiguous items back to your researchers.
How do you handle confidential models and data?
Awana signs an NDA and a standard DPA before any access, background-checks every evaluator, and keeps all work inside your environment, under your access tiers and redaction rules. Security and contracting.
How fast can a pod start?
First evaluators typically start in days, and most pods are staffed in about three weeks, including calibration against your gold set. On a coding-benchmark trial the first PR lands within hours of your accounts going live.
How is it priced?
The blind calibration round is free. After it, work is hourly or a fixed-price project: pilots are fixed-price, dedicated pods and surge capacity are billed hourly per evaluator, and a coding-benchmark trial is one fixed price for the week. You get a quote after the first call.
How is a pod different from a contributor marketplace or a student network?
Those are pools: many contributors, matched to tasks as the tasks arrive. An Awana pod is a fixed group of working senior software engineers who stay on your project, so calibration carries from one batch to the next. A pool fits volume and breadth. A pod fits code and agent work that needs production judgment.
How is it different from an expert network?
An expert network supplies credentialed specialists by the hour for deep domain questions. Awana supplies software engineers for code and agent work, as a standing pod inside your tools.
Can we compare you with our current vendor?
Yes. Run a pod beside your current vendor on the same tasks for a week and compare acceptance rate and rationale quality. The calibration round before it is free.
Does Awana sell datasets?
No. Awana does not build or sell datasets. Your tasks, rubrics and data stay in your environment.
Do you work with government and public-sector AI programs?
Yes. Awana is United States-led, and its leadership holds an active US security clearance. For public-sector work, Awana scopes staffing to the program's requirements, including US-based personnel where required.
What do you not do?
Awana does not run bio, chemical or nuclear risk testing. Dangerous-capability evaluation of frontier models is led by specialist evaluators, and Awana supplies trained capacity to them when they need it.
Scope your first pod.
Thirty minutes on the task, the rubric and your platform. You leave with a pod design and a calibration plan. The calibration round is free.
Security and contracting · Mercor alternatives · Also from Awana: Staff augmentation · Hire AI engineers · AI development · Private equity