Ограничения использования LLM для симуляции A/B-тестов

Обсуждали на одном из подкастов моё скептическое отношение к синтетическим людям. В дискуссию врывается Рон Кохави с теми же аргументами на примере разбора SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation.

I think there is value in using LLMs as a screening tool, and this paper is a good example. The tool could be used as a fast design-screening tool that makes predictions based on historical A/B tests, conventions, best practices, and folklore. It may work well against experiments similar to the history it has been trained on, but it is unlikely to work well for radical ideas (e.g. long-ad titles that I start my Maven course and book with). The title’s use of “Simulating” over-reaches, as it is impossible to establish causality from observational data without additional assumptions. LLMs are trained from historical data and are therefore not enough to simulate A/B tests without strong assumptions.

И

The system's greatest strength is acting as a "Shift-Left" tool in the design process. Before any engineering effort is spent coding a variant, SimAB can evaluate mockups to catch blatant usability flaws, confusing copy, or structural friction. As the authors note, it is an excellent mechanism to "kill bad ideas fast".

То есть да, что-то быстро проверить можно, но использовать как инструмент оценки, тем более численной, — это непонимание принципов работы LLM.