In short
An AI creative-testing claim is useful only when you can identify the model, the task, the comparison data and the decision it supports. Agreement with human labels, correlation with campaign outcomes, repeatability and causal lift are different forms of evidence. This guide shows how to examine each claim, including SaliencyLab's own, and turn the findings into a bounded purchase or pilot decision.
Research checked 16 October 2026. SaliencyLab publishes this guide and sells creative-review services and software. The checklist applies to our claims as well as other vendors'. It is an editorial evaluation framework, not an independent certification or a claim that the cited research tested SaliencyLab.
Start with the exact sentence
Consider the hypothetical claim “Our AI predicts winning ads with 90% accuracy.” Before debating whether 90% sounds impressive, identify what the sentence leaves undefined.
What is a win: a viewer's preference, a score above a threshold, higher click-through rate or an incremental purchase? What did the model see? Was the evaluation performed on ads it had not encountered during development? Did the data come from the platform and audience you intend to use?
Rewrite the claim in a form that can be checked:
Model/version ___ used inputs ___ to predict outcome ___ on ___ unseen examples from ___, evaluated on ___ using metric ___ against baseline ___. The uncertainty was ___ and the known exclusions were ___.
Blank fields are evidence requests. A vendor may have a legitimate reason not to release raw client data, but it should still explain the study design, scope and limitations. Confidentiality is not a reason to collapse several different validation tasks into one headline percentage.
Distinguish the evidence before comparing numbers
| Evidence type | Question it can address | What it does not establish by itself |
|---|---|---|
| Agreement with human labels | Does the model identify attributes similarly to the chosen annotators? | Whether changing those attributes improves sales |
| Held-out prediction | Does a specified model predict a named outcome on unseen examples? | That the relationship is causal or transfers to every platform |
| Test–retest stability | How much does the same workflow vary on a fresh run? | That a stable answer is correct |
| Real audience study | What did the recruited participants report or do under the study conditions? | Population representativeness or campaign lift without the relevant design |
| Synthetic interview | What hypotheses does a model generate under a persona and context? | The prevalence of those views among real consumers |
| Randomized campaign experiment | What difference did the assigned intervention make under the tested conditions? | A universal result for other audiences, campaigns or periods |
Do not rank a correlation coefficient, a classification accuracy and a human preference percentage as if they were the same measurement. First determine whether they address the same decision.
Seven checks for a vendor's evidence pack
1. Identify the evaluated component
A product can contain an attribute detector, attention model, composite score, benchmark percentile, simulated interview and a human recommendation. A study of one component does not automatically validate the others.
Ask the vendor to draw the boundary. If the study evaluates an attribute model against click-through rate, a headline about the entire platform predicting revenue reaches beyond that evidence. The output you plan to use should be traceable to the evaluated model and version.
SaliencyLab's methodology explicitly separates its platform-native attribute model from RoastIQ's five-KPI diagnostic composite, BuyerLens interviews and predicted attention maps. Buyers should preserve that separation when describing the results internally.
2. Inspect the split between development and evaluation
A held-out evaluation reserves data that were not used to fit or choose the model. Ask what was held out: individual rows, creatives, campaigns, brands or a later time period. Near-duplicate cuts from one campaign can make a random row split less demanding than a genuinely new-brand deployment.
Kapoor and Narayanan's research on data leakage explains how information crossing evaluation boundaries can inflate apparent performance and undermine reproducibility. Their work concerns machine-learning research broadly; it does not establish that any particular ad-testing vendor has leakage.
For a practical review, request the split rule, duplicate-handling procedure and what decisions were made after looking at the test set. Repeatedly tuning to a “holdout” makes it part of development. If your intended use is new campaigns for new brands, explain that use case and ask for evidence at that level.
3. Name the outcome and its source
An ad's public view count is not a measure of incremental purchases. Click-through rate can be relevant to a particular objective but does not by itself establish profitable acquisition. Observed performance also depends on distribution, audience, offer, media spend and campaign conditions.
Ask whether the target was measured directly, inferred from public data, provided by an advertiser or generated by another model. Record the collection date and whether missing values were excluded. If a validation target is itself synthetic, the study needs to be described as agreement with that target, not a direct measurement of people.
A benchmark pool has a different role again: it describes where a score sits relative to the selected reference set. A large pool can still be poorly matched to your market, format or category. Read the cohort and coverage rather than treating the total database size as your relevant sample.
4. Read uncertainty and the baseline together
Ask how performance compares with a sensible simple alternative. Would a majority-class rule achieve a similar classification accuracy? Would a basic model using platform and format explain much of the observed relationship? Which errors would change a creative decision?
Look for intervals or other appropriate uncertainty estimates, sample counts, subgroup coverage and an explanation of dependence between observations. Frames from the same ad and ads from the same campaign are not automatically independent examples. Reporting thousands of frames should not imply thousands of independently tested campaigns.
The REFORMS reporting standards provide a broader framework for transparent machine-learning research. Our buyer checklist uses the same practical concern—being able to inspect how a result was obtained—but is not the REFORMS checklist and should not be presented as compliance with it.
A rank correlation describes an ordering relationship. It is not a percentage of correctly predicted winners. A fitted-cohort estimate and a held-out estimate also answer different questions; their sample counts and intervals must remain attached to the evaluation that produced them.
5. Separate stable outputs from valid outputs
Run the same permitted input more than once when a pilot allows it, keeping the brief, settings and model version fixed. First establish whether the system produced a fresh result or returned a stored one. Reopening the same saved report does not test model stability.
Record variation in the decision-relevant outputs, not just a rounded headline score. A shift in the recommended edit may matter more than a small numerical difference. Decide in advance what level of variation would prevent the team from acting confidently.
For simulated audiences, fluent explanations can create an impression of evidential depth. Bisbee and colleagues' 2024 study of synthetic survey responses found that matching broad averages did not ensure matching variation or relationships in human survey data; prompt wording and time also affected responses. That study used political-opinion tasks and a particular model setting. It supports caution and task-specific validation, not a conclusion that every synthetic approach fails every creative task.
In SaliencyLab, BuyerLens follows a RoastIQ diagnostic and explores possible resistance in context. Its answers can help formulate questions for real people. They should not be counted as independent confirmation of the upstream diagnostic or reported as a representative audience percentage.
6. Match causal language to a causal design
A relationship between a creative attribute and an outcome does not demonstrate that editing that attribute causes the outcome to improve. Nor does a before/after comparison isolate the edit when targeting, spend, seasonality or the offer changed too.
Gordon and colleagues compared observational estimates with randomized results from 15 Facebook advertising experiments. In their setting, observational methods often did not recover the experimental effects, even with extensive controls. This is a reason to examine design when reading causal claims, not proof that every observational analysis is useless.
For a proposed creative pilot, define the intervention, assignment, outcome and analysis before delivery. A platform split test can compare versions under its design; incremental advertising effectiveness may require a different control or holdout. Ask the measurement owner which causal question the experiment actually answers.
7. Check whether the result transfers to your decision
A useful result in one language, platform or category can be weak evidence for another. List the differences between the evaluation and your work: audience, market, creative length, format, objective and publication period.
Ask for failure examples and unsupported uses. A vendor that clearly names an unsupported cohort gives you something actionable. “Works everywhere” without a corresponding evaluation makes the procurement question harder, not easier.
For high-consequence decisions, use independent human review and appropriate audience or campaign evidence. A diagnostic can still be useful for finding a subtitle mismatch or organizing an editing discussion even when the evidence is insufficient for a performance forecast.
Apply the checks to SaliencyLab's published evidence
As reviewed on 16 October, SaliencyLab's methodology reports held-out Spearman correlations of +0.33 for TikTok click-through rate and +0.40 for YouTube view counts for its platform-native attribute model. These are different outcomes evaluated separately. They are not percentages of correct predictions, evidence of causal lift or validation of the five-KPI composite.
The page separately reports fitted-cohort estimates with their own sample sizes and brand-clustered intervals. Do not attach those intervals to the held-out figures, combine the cohorts into a new sample, or extend the result to BuyerLens, attention heatmaps, sales or ROAS. Meta and unsupported cohorts remain directional.
The opening-frame evidence pilot offers an inspectable calculation using 18 AI-assisted observations. That makes the procedure easier to examine; it does not turn the observations into independent consumer research or a customer performance case. The native fictional QA sample similarly demonstrates an actual report format for a fictional creative, with known limits. Neither should be presented as a measured client outcome.
A reasonable use of SaliencyLab is to inspect source evidence, prioritize revisions and formulate a subsequent test. A requirement for a validated sales forecast, an independently proven BuyerLens replacement for human panels or guaranteed ROAS is outside the evidence described here.
Turn the evidence into a pilot decision
Download the creative-testing claim checklist (CSV). For each claim, record the source, date, evaluated model, outcome, sample, split, uncertainty, independent review and fit to your intended use.
Use three decisions rather than a manufactured confidence score:
- Suitable for a bounded pilot: the evidence supports the intended diagnostic use and the remaining uncertainty is acceptable for a small, reversible decision.
- Additional evidence required: a material field is missing, a cohort does not match, or a claim extends beyond the evaluated model. State exactly what would resolve it.
- Unsuitable for this requirement: the decision requires a capability the evidence does not establish, such as measured human recall or a causal sales forecast.
Before running the pilot, choose the assets and inclusion rules, record the version and define how people will judge the output. Include difficult examples, not just examples selected because the vendor performs well on them. Keep a simple alternative workflow for comparison. Record failures and inconclusive cases alongside useful results.
The final report should say what changed in the team's decision and what remains untested. “The review found a claim unsupported by the footage” can be a defensible outcome. “The tool increased sales” requires a different study.
For the commercial choice, see the three-ad budget guide. For confidential inputs, complete the procurement checklist. If an AI assistant supplied the shortlist, our recommendation audit shows why the cited passage needs checking too: a brand mention, an owned-site citation and an accurate recommendation are separate observations.