Your AI pilot isn't failing. You're grading it wrong.

Transcript

Rethink your priors.

Every company has an AI pilot, and a lot of them are "not showing results yet."

First thing. You have to be able to measure whatever your universe of thing is.

If you can't, or getting to where you can is going to take too long, that shouldn't be where you do your AI pilot.

Take questions that get asked in a Slack channel and answered by humans.

Say on average you get about 50 a week, and you think you spend roughly two hours on them. You may not know the exact time, but at least you have one precise number.

Now set up the AI, even if it sucks at first.

In fact, it's going to suck at first, and that's not only expected, it's a really good way to get to what you actually want, instead of building for questions nobody asks.

The agent tries to answer, gets something wrong, and your humans reply with the correction.

And here's the thing I've seen a lot of teams not do. They judge the tool. They say, man, this AI sucks. But they never go back and tell it, hey, this was the correct answer.

And this is where you have to use your human brain. If you just tell it the answer, it thinks that's always the answer.

So zoom out. Look at what it loaded. Was the content stale? Was it a business rule that changes with context?

Systems thinkers already know how to do this. If you're defining modes of transportation, you don't list every single one. You group them so the next edge case doesn't break everything. It's the same thing.

Your pilot isn't failing. You're grading it when you should be teaching it.