Skip to main content
search
Visionary Voices.

Use AI Agents, Yes, but Keep the Human Review.

As seen in MINDshift.

Paul Wash
Senior Tech Director, RGP

When a team wants to use AI to review changes to contract language, Paul Wash begins with a question. Can a model tell whether two passages mean essentially the same thing?  

That decision gets at a distinction Paul draws in this Visionary Voices conversation. AI can help an engineer write the code that runs a workflow. Giving AI a role inside that workflow is a separate decision. For CIOs, the question is where a business process calls for interpretation and how the team will know whether the model is doing that work well.

“I can’t tell you how many times I’ve seen companies basically install systems before they actually know what they’re supposed to do,” Paul says.

Understanding the Work

For a contract review project, Paul’s team used a low-code tool to test whether models could do the work the client needed. The client already had systems and procedures built up over time. His team built agents to help with part of the review and kept people in charge of reviewing the results. 

“Keep the human in the loop, keep control of what you’re doing, keep that human review,” Paul says.  

The people who know the process can tell when an interpretation fits the business and when it misses something. As Paul points out, some of what those people know has to be captured and given to the model before an agent can do its job. 

“Don’t just go in and build an entire agentic workflow. Focus on the parts of it,” he says. “Decouple the systems. Keep everything separate, the tasks that you’re doing.”  

In the contract-review project, that means testing whether the model can interpret a change in language while people still review the results. Once the team knows how each piece works with the context it’s given, Paul says, it can “string them together with an agentic flow.”  

And someone has to review the code, test it and see how it behaves. For Paul, AI can generate thousands of lines of code he might once have spent a day or two writing.  

“But I still have to piece those components together,” he says. “That observability of what I build is actually more crucial than the code itself.”

I’ve seen companies install systems before they know what they’re supposed to do.

Creating Cohesion in Experimentation

Paul encourages his teams to experiment, but not in isolation.  

“I totally think that there should be innovation, but I think pockets of innovation are dangerous because then there’s no coordination between the different teams,” Paul says.  

He describes teams building something exciting that becomes outdated or goes unused within months. Without coordination, what they learned can disappear with it. He also raises the questions an experienced engineer should ask when a prototype starts handling company data. “Is this legit code? Is this done well? Is this within our standards?” 

Paul sees a role for technology leaders in coordinating efforts and deciding which prototypes merit further investment. 

“Creating cohesion to me is also making sure there are common standards and platforms to be used,” he says.

He traces that view to a shift he’s seen in technology. Companies once tried to put everyone on the same systems. Over time, they gave divisions more freedom to build what they needed. That let teams solve problems closer to the work, but it also meant data and systems ended up scattered across the business. His answer is to agree on the standards and types of platforms teams use while giving them room to build for their own needs.

Creating cohesion to me is making sure there are common standards and platforms to be used.

The Bottom Line

When an AI model does work, such as interpreting contract language, the human team has to decide what counts as an acceptable result and how often the model needs to get it right. A successful demo can show that an agent can do the work once. Before relying on it, the business needs to know how consistently it performs. 

Paul puts the question plainly. “How do I validate that the AI reasoning is aligned with the context of the business and with expectations?”

Visionary Voices is a segment of RGP’s LinkedIn newsletter, Mindshift. Each month we highlight a unique futurist who challenges us to think differently and to drive innovation. Mindshift also contains valuable research and curated content.

Privacy Preference Center
RGP logo

When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and change your default settings.

Strictly Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

Functional Cookies

These cookies enable the website to provide enhanced functionality and personalization. They may be set by us or by third party providers whose services we have added to our pages. If you do not allow these cookies then some or all of these services may not function properly.

Performance Cookies

These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site. They help us to know which pages are the most and least popular and see how visitors move around the site. All information these cookies collect is aggregated and therefore anonymous. If you do not allow these cookies we will not know when you have visited our site, and will not be able to monitor its performance.