AI Use case - NLP
From theory to practice
Last year, Dominick Tonnarini joined my podcast, at the suggestion of my friend Soneel Choraria, to discuss real-life AI use cases in the internal audit profession.
The episode can be found here: https://tinyurl.com/yy7em8a4
Let me take that a step further with an example from my company.
Last year, one of our Operations leaders reached out to our team about a concern regarding a high-frequency error that was causing lost sales. (Sorry, I can’t disclose more details about the issue since internal auditors from competitors might read this post - lol.)
We pulled IT ticket data and, to our surprise, found over 100,000 tickets within the last year and a half related to the issue.
Tickets were categorized using drop-down options and included text data describing the issue and its resolution.
We could have relied on the categories alone without digging deeper for root causes.
But that’s not what internal auditors do, right?
Of course not.
Our next step was to identify common root causes and resolutions based on text data.
We worked with our data science team to create a plan for clustering the tickets based on certain keywords.
Clustering is a machine learning technique that groups similar data points.
In this case, the data points were keywords. Our final goal was to identify the most common root cause and the most common resolution for the errors.
It was far from an easy project.
At first, we tried Python.
It did not work well.
Next, at the suggestion of our data science team, we used a large language model (LLM) to perform natural language processing on the text data.
We spent a significant time on prompt engineering - believe me. Pro tip: Check out this website for prompt engineering ideas:
https://www.promptingguide.ai
At the end, between our human intelligence and the LLM’s help, we identified seven unique features to test for a cause-and-effect relationship.
Important to note: Our team identified five features, while the LLM identified two.
I believe we would not have identified those two features without the LLM.
We engaged with our service operations and network teams for further analysis.
To our frustration, we didn’t find a smoking gun. However, we identified a component likely causing issues. After consulting with our IT team, we discovered that this component was at its end-of-life stage, and its patches were probably contributing to some, but not all, of the problems.
All in all, it was a learning experience for our team.
Key takeaways from the experience included:
Natural Language Processing: The number one lesson is that natural language processing is a powerful tool for your internal audit toolkit. I recommend using LLMs in this space.
Probabilistic Thinking: The fact that we didn’t find a smoking gun highlights the importance of shifting our mindset as internal auditors. Sometimes, we need to look at the world through a probabilistic lens versus a deterministic lens. Believe me, I really wanted to find a smoking gun. However, in some areas reducing the likelihood of a “bad event” is all you can do. You won't find the smoking gun.
Human and Artificial Intelligence: Last, but not least, human intelligence is still far stronger than artificial intelligence. The LLM helped us quite a bit, but without our insights and the input from stakeholders, we wouldn’t have identified what we did.
How about you, how have you used AI in your practice?