Enterprises: Beware Of ‘Coherent Nonsense’ When Implementing Generative AI

Generative AI — using machine learning or other models (e.g., large language models) for creating novel text, images, sounds, etc. — is firmly in the spotlight as one of the fastest-developing areas of technology today, especially with the recent release of OpenAI’s ChatGPT. Earlier this month, we saw thousands of people experimenting with prompts and questions to have ChatGPT do tasks as varied as outlining a story or checking code for bugs. But despite the usefulness and accuracy of many of the results, ChatGPT and other large language models that generate text can easily generate coherent nonsense instead. Enterprises should be excited about experimenting with and integrating generative AI into the workflows of employees — but this excitement should not distract from the need to maintain the transparency and governance that currently exist around content generation processes and AI applications.

Those who use generative AI can inadvertently generate coherent nonsense either maliciously (e.g., using Galactica to generate racist or anti-Semitic research papers) or accidentally (e.g., when a user asks for content that they aren’t an expert in and thus can’t verify the information’s veracity). Both pose a risk to enterprises if such content makes it into production or any live environment. It will be crucial for enterprises adopting this technology to appropriately vet content and distinguish between something that looks correct on the surface versus something that is right in its meaning. To provide a human development analogy, we’ve reached the “late childhood” stage of generative AI systems. These systems can string together words convincingly and create logical arguments, but you can’t be sure if they’re just making things up or only telling you what you want to hear.

Generative AI Has Enormous Potential, But Manage It Like Any Other Enterprise Application

It’s important to remember that if generative AI makes things easier, it also makes doing bad things easier. The challenges created by coherent nonsense means that we need, more than ever, to maintain proper enterprise workflows and governance around AI-generated content to ensure that it has the accuracy and integrity that one would expect from any other enterprise software. While orchestrating enterprise processes with generative AI may feel onerous — especially given how quickly we can generate AI text and images — it will help teams avoid getting their fingers burned by releasing something into production that may have massive unintended consequences (e.g., Meta’s recent results with its controversial Blender AI bot). The risk goes beyond reputation — one could easily see a user asking ChatGPT for health advice and getting back suggestions that would make them sicker if there were no controls or guardrails on the use of the model.

Consider These Questions As You Begin To Experiment With And Adopt Generative AI Applications

As enterprises begin to explore this new universe of possibilities, there are many questions that persist about its application. Consider these core questions first:

• Did the training data come from a credible source? Is it likely to be correct?
• If an external partner trained the model, how will you audit the data sources to ensure that you have identified possible biases and confounders in the data?
• Does the solution understand context? Will it be able to understand new questions in reference to previous questions? Can different answers be given by understanding who a user is?
• Does the solution include an audit trail, chain of custody, or other indicators that it is generative content?

The Future Of Generative AI Is Bright — But Don’t Be Blinded By All The Possibilities

The future of generative AI is an ever-expanding universe of use cases. Every single part of the enterprise can reap benefits from this technology — from the contact center to marketing, product development, content generation, human resources, employee support, and customer experience. We have only begun to scratch the surface of what’s possible with these models today — and the pace of model development is delivering richer capabilities all the time. Enterprises must begin exploring these capabilities today — and should construct a strategy around how generative AI will impact their company from both the top down and the bottom up.

This post was written by Analyst Rowan Curran and it originally appeared here.