The Pros and Cons of AI for CRO Development

It's hard to find a corner of the tech world that generative AI hasn't reached. "Should we use AI here?" has quietly become "where shouldn't we?", and AI code has quickly evolved from a joke into standard industry practice.

CRO isn't building a website from scratch or maintaining a large application. We look to make carefully targeted changes or feature additions to an existing site to test a hypothesis. That matters because it changes which of AI's strengths and weaknesses apply. To understand why, it helps to start with an overview of how these models work.

A large language model is at its core a very complex predictor of the next piece of text. It trains on an enormous amount of data and learns the patterns within it. When prompted, it doesn't think, look anything up or reason from principles in the way a person does (despite how the outputs of most recent models may appear). It generates the most plausible continuation.

Firstly, this means an LLM is great at things it has seen many times before. Common coding patterns, DOM manipulation, standards and structures. It reproduces what it's seen fast and effectively.

Second, an LLM cannot produce something truly novel. It interpolates between things it has seen. It will confidently generate code or rationale that looks right when it isn't, and it will confidently say it's fixed something when it very clearly hasn't. This is because it's optimizing for plausibility rather than truth, and it can be very dangerous when applied to coding.

A typical test is a relatively small, standalone piece of code. It doesn't need to understand or integrate with a massive existing codebase. This means CRO code largely avoids the problems that plague AI on big software projects and the pattern matching of LLMs can work well.

Smaller tests can be scaffolded very quickly and it makes larger tests more feasible as the time cost of building them drops. Reducing build time and unlocking more sophisticated experiments can compound into a serious competitive edge.

This doesn't mean AI is a free win for CRO development, however. The same features that make these tests easier to complete also raise the stakes when something is wrong.

Stepping back from the code itself for a moment, there's a broader limitation worth noting: LLMs can't generate genuinely new ideas. Asking for test ideas tends to give the generic best practices everyone is already running. These can be helpful, but we know each website's audience behaves differently. That makes data-backed insights from human experts, grounded in real experience, more important than ever in a more competitive world where optimisation is increasingly common. The original thinking is exactly the part AI can't supply.

Back to the code, it ships straight to real users, often on the most important funnels. That means it has to be extremely high in quality and durable: resilient across browsers, screen sizes, devices, and the edge cases real traffic will throw at it. AI will happily produce code that works in the obvious case and quietly fails in others. It's important that a confident-looking snippet isn't shipped without the scrutiny it deserves.

Finally, websites are visual and contextual, and LLMs are fundamentally text-based. A page is a complex mix of layout, markup, hierarchy, spacing, and visual appearance. An AI working from code alone struggles to take all of this in, especially for visual-intensive tasks. It doesn't really know what it's doing, because it isn't actually seeing the rendered page like a human would. This is a significant limitation, though MCP tooling is starting to address it, providing richer webpage context at the cost of data size, tokens, and time.

CRO development turns out to be unusually well suited to AI in some respects and unusually exposed in others. The work is small-scale and pattern-heavy, which plays directly to the model's strengths and delivers real gains in pace and ambition. But it ships to real users on critical funnels, so must be used with extreme caution. CRO rewards original thinking models often can't produce, and it lives in a visual medium which a text-based system only partly grasps.

The teams that win with AI will lean on it for what it's good at while keeping human judgment firmly in charge of ideas and quality.