Human evaluators for RLHF, rubric grading and agent trajectories.
Awana provides human evaluation for LLMs through senior software engineers who work inside the evaluation harness you already run: preference and RLHF grading, blind rubric grading and agent trajectory review, calibrated against your gold set. It is for AI labs and enterprise AI teams. It is not a data vendor: Awana does not build, broker or sell datasets, and every output is your IP.
Sample · invented numbers
Trial read-out: coding-benchmark pod, week 1
- 42tasks shipped
- 86%accepted on first review
- 86%agreement with your gold set, floor 80%
- Engineer A16 tasks · 94% accepted first time · 91% agreementKeep. Strongest rationales in the pod.
- Engineer B15 tasks · 87% accepted first time · 88% agreementKeep.
- Engineer C11 tasks · 73% accepted first time · 76% agreementBelow your floor. We swap this seat within the week.
Human evaluation for LLMs, inside your pipeline.
Your tasks, your rubric, your harness. The engineers grade and write; the outputs stay yours. This is one of Awana's LLM evaluation services; a full adversarial program runs as AI red teaming services.
Pairwise comparisons and rankings with written rationales, in your queue
Pairwise · rankingScoring model outputs blind against your rubric
Your rubricGrading tool calls, multi-step plans and computer-use runs
Multi-stepAdversarial prompts and attack runs under your safety policy
Your policyWriting and tightening reference answers and grading guides you own
Your IPNative-level grading in English, Spanish and Portuguese
EN · ES · PTCalibrated against your gold set before live work.
- Four gatesinterview, take-home, English, references
- Matchedengineers picked for your task
- Blind calibrationgraded against your gold set
- Onboardingyour harness, rubrics and access
- Live cellagreement tracked every batch
Inside your perimeter, under your access rules.
Engineers work in your Slack, GitHub and internal harness. One accountable lead answers for each cell.
A cell, inside your environment
- Your Slack, GitHub and internal harness
- Your access tiers and redaction rules
- Your SSO or OAuth, least privilege, logged sessions
- Access removed at offboarding
- A background check on every engineer before access
- NDA and standard DPA before any data is shared
- PII masked before human review
- No datasets: we do not build, broker or sell them
Senior engineers beside the vendors you already use.
Prove the signal first. Then fund the cell.
Free blind calibration round
Vetted engineers complete a small sample of your real repo-level tasks or red-team prompts, blind.
Paid one-week trial pod
Runs alongside your existing vendors on the same tasks, so you can compare quality and velocity.
Dedicated pod retainer
A multi-engineer cell on your work, funded from your post-training or evaluation budget.
The calibration round is free. The trial and the retainer are quoted after the first call.
01What human evaluation work do your evaluators do for LLMs?
Preference and RLHF grading: pairwise comparisons and rankings with written rationales, in your queue. Rubric grading: scoring model outputs blind against your rubric. Agent trajectory review: grading tool calls, multi-step plans and computer-use runs. They also write and tighten the rubrics and gold items you own, and grade in English, Spanish and Portuguese.
02Does Awana build or sell datasets?
No. Awana does not build, broker or sell datasets. The engineers work inside your environment, and all code, rubrics and evaluation outputs are your IP.
03How does a cell fit beside the data vendors we already use?
It plugs into the harness and vendors you already have. Platforms and networks keep the volume and breadth work; the cell takes the code and agent tasks that need senior engineers. A trial can run beside your current vendor on the same tasks.
04How does Awana calibrate evaluators?
Every engineer passes the four vetting gates (a live technical interview, a take-home built for the role, an English screen and reference checks), then a blind calibration test against your gold set before touching live work. When your rubric changes, Awana re-calibrates the cell before the next batch.
05How does Awana measure quality and agreement?
Awana tracks inter-rater agreement on every batch against a floor you set, runs second-pass review on a sample, and logs ambiguous items back to your researchers so the rubric gets sharper over time.
06How does Awana handle confidential models and data?
Awana signs an NDA and a standard DPA before any data is shared, background-checks every engineer before they touch your systems, and keeps all work inside your environment. PII is masked before human review.
07How do we start?
With a free blind calibration round: vetted engineers complete a small sample of your actual repo-level tasks or red-team prompts, blind. Next comes a paid one-week trial pod beside your existing vendors, then a dedicated pod retainer.
08How is an evaluation cell priced?
The calibration round is free. The one-week trial pod is paid, and the dedicated pod retainer is funded from your post-training or evaluation budget. Awana quotes both after the first call, and adds engineers for launches or eval pushes when you need them.
More questions are answered in the text version of this page.
- A cell designTasks, harness access, domain skills and languages
- A calibration planSample tasks, gold set and agreement floor
- The path and a quoteFree calibration, a paid one-week trial pod, then a dedicated pod retainer