In Tackling Near-Term AI Risks, Old-School Monitoring and Filtering Have Value

I had a different essay prepared for this week. But I attended a conference yesterday, heard concerns about AI that match concerns I’ve heard at other conferences this past summer, and wanted to address a couple of those. Note: this post is about immediate risks of AI, not longer-term concerns. I address longer-term AI risk in other essays.

First, I polled the audience at two risk conferences where I spoke and asked what they saw as immediate risks of AI. In addition to algorithmic bias (absolutely a problem right now, today), both audiences mentioned intellectual property (IP) exposure or loss.

Old-School DLP, New Use Case?

That’s interesting, because data leakage has been a problem for companies for decades if not centuries, and we have many and varied (though imperfect) solutions for it. Take data loss prevention (DLP) for email messages: companies routinely scan emails for personally identifiable information (PII)—stuff like social security numbers, names, addresses, and birth dates)—as well as proprietary information (aka IP). Emails that fail the scan don’t get sent. Is it perfect? No. Does it reduce risk a lot? Yep.

Why can’t companies eventually apply a similar DLP approach to data sent into AI systems? In my view, they totally could. Data is coming from company systems, so it can be scanned before it ever even reaches an AI system as a prompt. Once it gets into the AI system, that’s when it can become difficult to extract the data. But before it gets there? Filtering for PII and other flags is an opportunity to stop much of it: a preventive control.

Taking a broader view, many “good old operational risk controls” have relevance as companies consider how to integrate AI into their workflows. Yes, other AI-specific techniques and controls will become necessary as AI advances. But companies have valuable tools for reducing risk already.

Monitoring for Uncertainty

Second, most people know by this point that large language models (LLMs) hallucinate: they emit falsehoods sometimes. This is almost an inherent property of how LLMs work, although as models advance, they will likely hallucinate less often.

A short explanation: each token (word or portion of a word) is generated by calculating what the most probable next token should be, based on the previous tokens, but that’s not guaranteed to be correct. In extremely simple terms, the most probable next token may be close to the first runner-up token (say, 45% probability versus 43% probability), or the most probable next token may be a clear winner (63% probability versus 22% for the first runner-up), but you do not see this nuance in the output. (If you want the long explanation, Stephen Wolfram has an excellent novella-length post on how ChatGPT works. It’s a good read!)

I’ve had a few conversations about this in the past few weeks, at conferences and in calls, and my thoughts remain as follows: it seems to me that it might be possible to track this type of nuance via separate monitoring software. For example, if the probabilities for each next token could be reported to and stored in monitoring software as output was generated, is there a way for monitoring software to cut in? Perhaps an option for probabilistic nuance to be reflected visually (bold, grayed out, etc.) in the text output? Or, after output is generated but before it’s displayed to the user, could a filtering layer of fact-checking software vet citations, references, and facts against canonical databases or encyclopedias (kind of a DLP approach but for filtering falsehoods, not PII)? That might be too costly for some use cases but worthwhile for others.

Do I have the answers to this and know exactly how to do it? Absolutely not. Do I think there are under-explored potential solutions based on old-school concepts of monitoring and filtering? Absolutely yes.

Thanks for reading! Next week I have a great essay on why selling is an undervalued skill in risk management, based partly on a presentation I gave years ago but updated for today’s environment.