The Definitive Guide to Large Language Models and High-Performance Marketing Content

As we pass through the “trough of disillusionment” in generative AI, enterprise marketers are beginning to understand the inherent limitations of using today’s probabilistic Large Language Models (LLMs) with regard to trust, scale, and performance for marketing engagement. LLMs were built to be generally good at many different tasks. In reality, AI models leveraging symbolic logic may be necessary to counter the challenges LLMs face with trust and scale. Performant short-form text content is the last remaining vector for marketers to leverage AI to improve open and conversion rates through digital channels. This paper firstly demonstrates that LLMs alone aren’t well suited to solve the needs of the modern marketer due to their inherent limitations. Secondly, it proposes an alternative solution for performant content generation at scale.

Introduction

In the rapidly evolving landscape of artificial intelligence (AI), few innovations have captured the attention of both tech enthusiasts and the public as much as Large Language Models (LLMs). These algorithms have dominated the AI news cycle and found their way into the mainstream media, leading to a mix of justified excitement and unjustified speculation. Almost everyone in the AI community agrees that LLMs are not just a passing trend. LLMs are here to stay. Amara’s Law states that people will tend to overestimate the impact of a technology in the short term and underestimate the impact in the long term. Only through understanding the inherent limitations of a novel technology can we accurately understand (and predict) what kind of impact it may have.

LLMs are ushering in a new era of marketing innovation. This is one of the first use cases that emerged after the release of OpenAI’s GPT-3 in 2020 and continues to be one of the most exciting applications. Understanding this space is not an option but a necessity for any modern and forward-thinking company.

Some companies will proceed with skepticism and caution, perhaps banning LLMs entirely. These companies may be forced to hire additional staff to enable content generation at scale. There are more viable and competitive long-term strategies than this. Most companies have accepted that they will increasingly rely on generative AI for content over the coming months and years. According to Gartner, by 2025, 30% of outbound marketing will be machine-generated, up from around 2% in 2022.

Companies adopting generative AI are faced with two options:

Developing with Large Language Models (LLMs)

Companies with internal development resources may wish to investigate building an in-house solution. In theory, this might be more cost-effective and fit for purpose. There are many tools and platforms available for developing with LLMs. For example, OpenAI has an easy-to-use API, and several open-source models, such as Meta’s Llama, are available for commercial use.

The above are inherent limitations of LLMs in general. There are also practical development and engineering limitations:

Fine-tuning with training data

Fine-tuning Large Language Models (LLMs) involves additional training of pre-existing language models for specific tasks or domains. In other words, instead of training an LLM from scratch (with a price tag in the millions), fine-tuning builds on the parameters already learned by the base (or “foundation”) model. This is done through a process of supervised learning. The first step is to create a training set. This is a set of input examples and expected (or acceptable) output. The training algorithm adjusts the weights of the underlying model in response to the new training data. It is important not to confuse the machine learning definition of fine-tuning with the dictionary definition – in machine learning, to fine-tune does not simply mean “to improve.”

Some companies use their proprietary data to fine-tune an LLM. They hope this will improve the output quality and build a defensive moat between themselves and their competitors. There are some areas where fine-tuning is known to be helpful. For example, it can significantly improve classification task accuracy when specific domain expertise is required (e.g., processing legal documents). However, the process is not as straightforward or effective as one might hope for text-generation applications.

The quality of training data is a key variable for the success of fine-tuning. Significant manual effort is often required to ensure the training data is suitable. The training data should be:

Generation

A training set is required to fine-tune an LLM. You can’t simply use examples of campaigns that have performed well in the past. Many factors contribute to a campaign’s success, and the wording of a language is just one of those factors. In fact, the same content can have differing performance depending on when it is delivered, in what channel it is delivered, and to whom it is delivered. Here are some of the factors that impact the performance of a piece of content:

Therefore, raw engagement metrics do not necessarily correlate with the quality of the content. The solution is to train on split test data. In this case, the message variants are sent to different random subsets of an audience simultaneously. Since there is only one independent variable, any difference in performance can be attributed to the content variations themselves.

One way to frame the high-performance generation problem is as machine translation. Instead of translating between two languages (e.g., English and Icelandic), the goal is to re-write the input into a new version expected to perform better. As with translation between languages, the training data is language pairs: one variant with poor performance and a corresponding variant with high performance. For example:

These results are not surprising when you consider the training data. Consider this example again:

Even though the underlying messages between the input and output are consistent (“shoes are on sale”), the high-performing output contains additional words and phrases to add context and flavor (without fundamentally changing the meaning). By structuring the training data in this way, the system has “learned” that you can add language to improve the performance of the content. However, it cannot distinguish between permissible and impermissible content additions.

This problem cannot be solved by increasing the amount of training data. Other third-party results also corroborate these findings, with recent research finding 3% to 27% of results include hallucinations.

There are other ways to structure the training data. For example, you can use pattern matching and named entity recognition (NER) to ensure the input and output have the same information content. However, in this case, the system learns that only small tweaks can be made to improve content performance. In reality, tweaks are unlikely to impact the performance of the language. Alternatively, the training data can be structured so the system learns commonplace expressions. In this case, the output is highly generic and suffers from a lack of diversity.

Performance prediction

The experiments in the previous section demonstrate that fine-tuning is not a promising approach for coaxing LLMs into generating high-performance content. The solution is to logically separate the problems of generation and performance (more on this below). However, perhaps we can fine-tune an LLM to learn to distinguish high-performing content from low-performing content. This would be an alternative to a smaller and dedicated machine learning model, with the motivation being that perhaps an LLM has an implicit notion of language and performance that would give it an edge over smaller models.

We ran an experiment to compare the accuracy of a fine-tuned LLM against our task-specific performance prediction engine. The task-specific engine uses an attention-based Long Short-Term Memory (LSTM) neural network.

Analysis

Content generation and performance optimization are distinct and orthogonal problems. Our experiments suggest that conflating these problems in a single LLM leads to a higher hallucination rate. This is because the model cannot distinguish between core content (what to say) and stylistics (how to say it). The former is fixed, while the latter can be tailored to suit an audience. By trying to solve both problems simultaneously, the system is prone to maximize performance by changing the message’s meaning. BCG has done a similar analysis highlighting the challenges of fine-tuning your own model and how responses are bound by the model’s capability and original dataset.

There is another risk when content and performance are intertwined. A well-known principle in business is Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. This manifests itself in AI-powered content generation in two ways:

• It is usually possible to improve one metric at the expense of another. For example, if the AI system is designed to maximize open rates, this can be done at a cost to click rates by using spammy and misleading subject lines. An example would be a subject line “Congratulations, you’ve won a $100 gift card!” with the body of the email containing a message along the lines of “Gotcha! While you’re here, check out these deals”. A strategy like this would achieve high open rates in the short term. However, it would also alienate and anger an audience. This is an extreme example for illustration, but it demonstrates that there is a risk when an AI system optimizes towards a single metric without checks and balances in place.
• Another risk of treating every campaign in isolation is that it can lead to repetition over time. The system might decide on the “best” content and use it repeatedly. Eventually, this will lead to ad fatigue and a disengaged audience.

Best Practices

Here are some best practices to keep in mind when developing a system to maximize the performance of marketing content:

Conclusion

We are in the early days of a technological revolution driven by AI. However, many open questions exist about how to use this powerful new technology. There is no shortage of vendors who are offering impressive-sounding products. However, many of these are little more than a thin user interface in front of technology they do not own (or even fully understand).

When developing in-house solutions, it may sound compelling to say: “We have data we have been collecting for years. Let’s feed that into an LLM.” Firstly, we have demonstrated in this white paper that this isn’t a fruitful approach to the problem. Secondly, and more importantly, this isn’t the correct way to approach R&D. There is a lot of hype around AI, and a fear of missing out is driving many companies’ technical decisions. They are looking for any way they might be able to integrate LLMs into their tech stack. However, this is backward. The correct approach is to start with a well-defined, valuable, and pre-existing problem. Next, evaluate a range of possible solutions. LLMs will be integral to solving complex problems like generating high-performing marketing content. However, the solution will also require various other technologies, such as natural language processing, reinforcement learning, optimization algorithms, and heuristic rules. Sequoia Capital declared that the next generation of performant technologies will use foundational models as a piece of a more comprehensive solution rather than the entire solution. Embracing a holistic approach ensures the effective utilization of AI in solving complex challenges.