Evaluate agreement before claiming reliability.

Freeze the task and evaluation material

Define each attribute, allowed label, missing-data rule and the evidence an annotator should inspect. Save the codebook version, model identity, prompt hash and decoding settings.

Keep training examples and held-out evaluation creatives separate. Record how the sample was selected and which markets, formats and platforms it covers. Do not set the desired agreement percentage in advance.

Collect independent annotations

Have reviewers label the same eligible items independently before discussing disagreements. Record missing and not-applicable cells explicitly. Preserve original annotations when adjudicating differences.

Count the actual comparisons used by each metric. The number of ads multiplied by the number of attributes is not automatically the number of usable paired ratings.

Report limits beside the result

Report the metric definition, eligible denominator, label distribution and uncertainty. Select an agreement measure appropriate to the labels and study design; document that choice before interpreting the results.

Break down performance by attribute and relevant cohort, with enough examples to explain disagreement. Raw agreement, chance-adjusted reliability, model repeatability and outcome prediction answer different questions.

Release evidence that can be checked

Publish a versioned method and a permission-compatible replication record before calling a study completed or a paper published. A proposed journal or planned dataset release is not publication evidence.

Agreement on visible attributes does not validate every RoastIQ KPI, synthetic BuyerLens response or campaign forecast. Keep those claims separate and retain human review for consequential decisions.

Continue the review

Use the Hook Timer to mark creative moments, read the methodology and evidence limits, or return to the industry review guides.