Human evaluators for RLHF, rubric grading and agent trajectories.

Awana provides human evaluation for LLMs through senior software engineers who work inside the evaluation harness you already run: preference and RLHF grading, blind rubric grading and agent trajectory review, calibrated against your gold set. It is for AI labs and enterprise AI teams. It is not a data vendor: Awana does not build, broker or sell datasets, and every output is your IP.

Free blind calibration round · UTC-3 to UTC-6, full overlap with the US workday

50 engineers on AI-training and evaluation work for a frontier AI lab in three monthsUnited States-led, with senior engineers across Latin AmericaRoughly 1 in 40 engineers make it through our vetting

Sample · invented numbers

Trial read-out: coding-benchmark pod, week 1

  • 42tasks shipped
  • 86%accepted on first review
  • 86%agreement with your gold set, floor 80%
  • Engineer A16 tasks · 94% accepted first time · 91% agreementKeep. Strongest rationales in the pod.
  • Engineer B15 tasks · 87% accepted first time · 88% agreementKeep.
  • Engineer C11 tasks · 73% accepted first time · 76% agreementBelow your floor. We swap this seat within the week.
A sample from a coding-benchmark pod, shown for the format of the read-out you receive after the paid one-week trial pod. Full sample read-out (PDF)

Human evaluation for LLMs, inside your pipeline.

Your tasks, your rubric, your harness. The engineers grade and write; the outputs stay yours. This is one of Awana's LLM evaluation services; a full adversarial program runs as AI red teaming services.

Preference and RLHF grading

Pairwise comparisons and rankings with written rationales, in your queue

Pairwise · ranking
Rubric grading and evals

Scoring model outputs blind against your rubric

Your rubric
Agent trajectory review

Grading tool calls, multi-step plans and computer-use runs

Multi-step
Red-team sprints

Adversarial prompts and attack runs under your safety policy

Your policy
Rubrics and gold items

Writing and tightening reference answers and grading guides you own

Your IP
Multilingual evaluation

Native-level grading in English, Spanish and Portuguese

EN · ES · PT

Calibrated against your gold set before live work.

  1. Four gatesinterview, take-home, English, references
  2. Matchedengineers picked for your task
  3. Blind calibrationgraded against your gold set
  4. Onboardingyour harness, rubrics and access
  5. Live cellagreement tracked every batch

Inside your perimeter, under your access rules.

Engineers work in your Slack, GitHub and internal harness. One accountable lead answers for each cell.

A cell, inside your environment

Tasks come in, engineers work them, one lead answers for the result, graded work goes back. The cell works inside your tools and access rules.
Your environment, your rulesThe access model you already use
  • Your Slack, GitHub and internal harness
  • Your access tiers and redaction rules
  • Your SSO or OAuth, least privilege, logged sessions
  • Access removed at offboarding
Nothing leaves your environment
Our side of the contractWhat Awana commits to
  • A background check on every engineer before access
  • NDA and standard DPA before any data is shared
  • PII masked before human review
  • No datasets: we do not build, broker or sell them
Code, rubrics and outputs are your IP

Senior engineers beside the vendors you already use.

Senior engineering signalSoftware engineers who grade AI output for clients today, each through the four vetting gates.
Beside your vendorsVolume and breadth stay with the platforms you use. The cell takes the work that needs senior engineers.
Same hoursUTC-3 to UTC-6, full overlap with the US workday, so calibration happens live.
No lock-inNo platform of ours and no dataset. Your harness, your rubrics, your IP.
DomainsSoftware engineeringCode reviewPythonTypeScriptMath and reasoningData and SQLAgentic tasksSafety red-teamingEnglishSpanishPortuguese

Prove the signal first. Then fund the cell.

Free blind calibration round

Vetted engineers complete a small sample of your real repo-level tasks or red-team prompts, blind.

Freebefore any commitment

Paid one-week trial pod

Runs alongside your existing vendors on the same tasks, so you can compare quality and velocity.

Paid trialone week, on your tasks

Dedicated pod retainer

A multi-engineer cell on your work, funded from your post-training or evaluation budget.

Retainerthe same engineers, month after month

The calibration round is free. The trial and the retainer are quoted after the first call.

The basics
01What human evaluation work do your evaluators do for LLMs?

Preference and RLHF grading: pairwise comparisons and rankings with written rationales, in your queue. Rubric grading: scoring model outputs blind against your rubric. Agent trajectory review: grading tool calls, multi-step plans and computer-use runs. They also write and tighten the rubrics and gold items you own, and grade in English, Spanish and Portuguese.

02Does Awana build or sell datasets?

No. Awana does not build, broker or sell datasets. The engineers work inside your environment, and all code, rubrics and evaluation outputs are your IP.

03How does a cell fit beside the data vendors we already use?

It plugs into the harness and vendors you already have. Platforms and networks keep the volume and breadth work; the cell takes the code and agent tasks that need senior engineers. A trial can run beside your current vendor on the same tasks.

Quality
04How does Awana calibrate evaluators?

Every engineer passes the four vetting gates (a live technical interview, a take-home built for the role, an English screen and reference checks), then a blind calibration test against your gold set before touching live work. When your rubric changes, Awana re-calibrates the cell before the next batch.

05How does Awana measure quality and agreement?

Awana tracks inter-rater agreement on every batch against a floor you set, runs second-pass review on a sample, and logs ambiguous items back to your researchers so the rubric gets sharper over time.

Security and access
06How does Awana handle confidential models and data?

Awana signs an NDA and a standard DPA before any data is shared, background-checks every engineer before they touch your systems, and keeps all work inside your environment. PII is masked before human review.

Getting started
07How do we start?

With a free blind calibration round: vetted engineers complete a small sample of your actual repo-level tasks or red-team prompts, blind. Next comes a paid one-week trial pod beside your existing vendors, then a dedicated pod retainer.

08How is an evaluation cell priced?

The calibration round is free. The one-week trial pod is paid, and the dedicated pod retainer is funded from your post-training or evaluation budget. Awana quotes both after the first call, and adds engineers for launches or eval pushes when you need them.

More questions are answered in the text version of this page.

Scope your calibration round.

Thirty minutes with Calvin Sedao, our CEO.

Book a call →
  1. A cell designTasks, harness access, domain skills and languages
  2. A calibration planSample tasks, gold set and agreement floor
  3. The path and a quoteFree calibration, a paid one-week trial pod, then a dedicated pod retainer