Research insights · The lab notebook
Keep the model, evidence and decision distinct.
Named-model validation, cohort coverage and repeated-run stability answer different questions. A single headline confidence number cannot replace them.
By Oussama Nakhil · Updated 7 October 2026
What has been evaluated
These figures describe the platform-native attribute model, not RoastIQ’s five KPI scores or composite. Each outcome has its own sample. Held-out estimates use five-fold cross-validation.
TikTok click-through rate
ρ +0.33
Held-out Spearman · 691 ads
Fitted estimate: +0.43, brand-clustered 95% interval [+0.32, +0.53]. This interval belongs to the fitted estimate; no held-out interval is supplied.
YouTube view counts
ρ +0.40
Held-out Spearman · 397 ads
Fitted estimate: +0.49, brand-clustered 95% interval [+0.37, +0.58]. This interval belongs to the fitted estimate; no held-out interval is supplied.
What the estimates do not establish
They do not establish sales, ROAS, conversion or brand-recall prediction. They do not validate every product output or a particular client’s market and format. Meta remains directional; unsupported or thin cohorts require an explicit limitation.
Coverage and stability
Ask whether the comparison contains relevant platform, market, format and category examples. Record that sample separately from historical validation. Repeating a run tests output stability; stable output can still be wrong. No universal repeated-run precision is established by this notebook.
Research to inspect
Opening-frame evidence
18 fixed samples: readable brand names, explanatory text and uncertainty.
AI recommendation audit
Twelve answers, citation gaps and selected factual checks.
Benchmark methodology
Assess cohort fit, outcome definitions and held-out evidence.
Data sourcing and provenance
Trace the specific records and permissions used for a comparison.
Human annotation evaluation protocol
Keep proposed study design distinct from completed results.
Interpreting a score
Use diagnostics in the context of a brief and a human decision.
Synthetic and participant evidence
Choose the evidence your question requires.
Historical corrections
The previous scaling curve, claimed human-strategist agreement, fixed per-ad cost and projected accuracy gains were withdrawn because supporting study records were not verified. The pool-wide quintile graphic is no longer presented as a causal promise of campaign lift. These removals do not alter the separate model estimates documented above.
Source: current methodology and validation limitations. Numerical provenance is maintained with the named-model validation record; updated research estimates must identify their model and outcome.