Share:
Use AI Agents, Yes, but Keep the Human Review.



When a team wants to use AI to review changes to contract language, Paul Wash begins with a question. Can a model tell whether two passages mean essentially the same thing?
That decision gets at a distinction Paul draws in this Visionary Voices conversation. AI can help an engineer write the code that runs a workflow. Giving AI a role inside that workflow is a separate decision. For CIOs, the question is where a business process calls for interpretation and how the team will know whether the model is doing that work well.
“I can’t tell you how many times I’ve seen companies basically install systems before they actually know what they’re supposed to do,” Paul says.
Understanding the Work
For a contract review project, Paul’s team used a low-code tool to test whether models could do the work the client needed. The client already had systems and procedures built up over time. His team built agents to help with part of the review and kept people in charge of reviewing the results.
“Keep the human in the loop, keep control of what you’re doing, keep that human review,” Paul says.
The people who know the process can tell when an interpretation fits the business and when it misses something. As Paul points out, some of what those people know has to be captured and given to the model before an agent can do its job.
“Don’t just go in and build an entire agentic workflow. Focus on the parts of it,” he says. “Decouple the systems. Keep everything separate, the tasks that you’re doing.”
In the contract-review project, that means testing whether the model can interpret a change in language while people still review the results. Once the team knows how each piece works with the context it’s given, Paul says, it can “string them together with an agentic flow.”
And someone has to review the code, test it and see how it behaves. For Paul, AI can generate thousands of lines of code he might once have spent a day or two writing.
“But I still have to piece those components together,” he says. “That observability of what I build is actually more crucial than the code itself.”
Creating Cohesion in Experimentation
Paul encourages his teams to experiment, but not in isolation.
“I totally think that there should be innovation, but I think pockets of innovation are dangerous because then there’s no coordination between the different teams,” Paul says.
He describes teams building something exciting that becomes outdated or goes unused within months. Without coordination, what they learned can disappear with it. He also raises the questions an experienced engineer should ask when a prototype starts handling company data. “Is this legit code? Is this done well? Is this within our standards?”
Paul sees a role for technology leaders in coordinating efforts and deciding which prototypes merit further investment.
“Creating cohesion to me is also making sure there are common standards and platforms to be used,” he says.
He traces that view to a shift he’s seen in technology. Companies once tried to put everyone on the same systems. Over time, they gave divisions more freedom to build what they needed. That let teams solve problems closer to the work, but it also meant data and systems ended up scattered across the business. His answer is to agree on the standards and types of platforms teams use while giving them room to build for their own needs.
The Bottom Line
When an AI model does work, such as interpreting contract language, the human team has to decide what counts as an acceptable result and how often the model needs to get it right. A successful demo can show that an agent can do the work once. Before relying on it, the business needs to know how consistently it performs.
Paul puts the question plainly. “How do I validate that the AI reasoning is aligned with the context of the business and with expectations?”