18 Apr 2017
What is: natural language generation (NLG)?
Saying (or writing) the right words in the right sequence to convey a clear message that can be easily understood by the listener (or reader) can be a tricky business.
For a machine, which processes information in an entirely different way than the human brain does, it can be trickier still.
Solving this issue has been the key focus of the burgeoning field of Natural Language Generation (NLG) for years beyond counting.
Natural language generation, a field which has made great strides of late, has begun to manifest in many areas of our lives. From our chats with Siri, Alexa, Google Assistant, and Cortana, to our interactions with the complaint departments and tech support of our favourite brands, NLG has begun to play an increasingly important role in our lives, and this trend shows no signs of slowing down any time soon.
So what exactly is natural language generation?
What is: natural language generation?
Human language is complex.
There are approximately 6,500 distinct languages spoken worldwide by earth’s 7.125 billion inhabitants. Of these, about 2,000 languages are spoken by fewer than 1,000 people. Compare those languages to Mandarin Chinese, with its 1.2 billion native speakers, and you begin to get a picture of the variance in the global language data available.
For our purposes, we are going to zero in on the English language and NLG efforts focused on English. Although English has comparatively small number of native speakers worldwide (estimated to be about 400 million), it is spoken as a secondary language by as many as 1.6 billion people worldwide. English also happens to be the native language of our humble Phrasee home here in Putney, southwest London, so there’s that.
According to The Oxford Dictionary, its 20-volume Second Edition contains:
“Full entries for 171,476 words in current use, and 47,156 obsolete words. To this may be added around 9,500 derivative words included as subentries. Over half of these words are nouns, about a quarter adjectives, and about a seventh verbs; the rest is made up of exclamations, conjunctions, prepositions, suffixes, etc. And these figures don’t take account of entries with senses for different word classes (such as noun and adjective).
This suggests that there are, at the very least, a quarter of a million distinct English words, excluding inflections, and words from technical and regional vocabulary not covered by the OED, or words not yet added to the published dictionary, of which perhaps 20 per cent are no longer in current use. If distinct senses were counted, the total would probably approach three quarters of a million.”
That’s a lot of words.
Natural language generation is complex
With so many variables to choose from, there are literally billions of ways that any series of words can be strung together to convey a message. Add the English language’s grammar rules (which are also complex), and you’ve got yourself a very interesting mathematical problem.
Natural language generation is the process of developing a learning machine capable of sorting through all these variables and putting them together into natural, human-sounding sentences, statements, or paragraphs without intervention from the handler.
To accomplish this massive feat, developers have leaned heavily on the computing principle of the algorithm. Algorithms are purpose-built computer programs capable of sorting through large sets of data and reordering that data according to a predetermined set of objectives. In the case of natural language generation, such algorithms focus on they way human language is structured and sort through all the available words in an attempt to artificially replicate human language patterns.
Tone of voice is complex
As Microsoft learned the hard way through its disastrous “Tay” chatbot experiment on Twitter, there is more to interacting with humans than simply producing structurally sound and grammatically correct sentences.
Tay, which took its learning cues from ongoing interactions with people’s Twitter accounts, began making outlandish and offensive statements publicly within hours of being set loose in the Twitterverse. The well-publicised event served as a reminder to brands that when it comes to natural language generation and marketing, things are complicated.
When speaking on behalf of a brand, particularly in the politically and socially charged times in which we are living, one must always watch what one says. This is just as true of NLG programs as it is for their human copywriting counterparts.
In a world where AI, machine learning, and natural language generation are being asked to share more and more of the brand marketing burden, brand voice/tone of voice will become a more important issue with each passing day.
Phrasee is complex
Here at Phrasee, we have been working in the field of natural language generation for the purposes of digital marketing for years. We have encountered first hand the problems associated with sentence structure, tone of voice, and brand voice that are a natural side effect of the endeavour to couple a developing technology like natural language generation with marketing language.
Natural language generation is indeed a complex problem, which requires equally complex solutions.
But, as we have learned through an exhaustive process of trial and error (and a whole lot of research and experimentation), delivering real, high performing results is entirely possible.