- AI evaluation
- Product delivery
- Responsible AI
How good is good enough? Setting the bar before building GrantIQ
- Role
- Product Manager
- Company
- Grant Instruments
- When
- Aug to Dec 2025
- Tools
- Microsoft Copilot Studio, SharePoint, Teams, NotebookLM
- on a 120-question test, target 80%
- 100%
- per answer
- 2-3 hrs → ~1 min
- people on the support team
- 15
Before
- Answers spread across hundreds of manuals and datasheets
- A simple question could take 2 to 3 hours
- No way to know whether an AI would make things up
After
- Answers in about a minute, inside the chat tool the team already used
- Answers link to the document they came from
- 100% accuracy on a 120-question test, before anyone relied on it
01The problem
Grant Instruments makes precision lab equipment, from water baths to shakers. Customers ask very specific questions: will this fit on my bench, will this accessory work with it, how cold can it go? The answers existed, but they were spread across hundreds of manuals and datasheets on an old internal server. With so many models and variations, the customer support team could spend 2 to 3 hours answering a single simple question. AI could find answers in seconds, but with products that differ by a few litres or a few degrees, an AI that makes things up would do more harm than good.
My job was to build an AI assistant accurate enough for the team to trust.
02What I did
- Proved it small first. I built a quick prototype on one of the hardest product ranges, where models differ only by tank size. Our lead product manager wrote 50 test questions, some deliberately tricky. It answered most of them, and when it didn't know, it said so instead of guessing. It did well enough that he joked he was worried for his job.
- Set the bar before building. I agreed the targets up front: at least 80% of answers exactly right on a 120-question test, and most answers pointing to their source.
- Sorted the documents first. I moved the manuals off the old server and organised them by product, so the AI was working from clean, current files.
- Built it where the team already worked. IT set up the platform; I built the assistant itself, inside Microsoft Teams. It only shows people files they're already allowed to open.
- Checked the cost. I estimated how much the team would use it and showed it fitted within a single licence pack, with room to spare.
- Wrote the rules for customers. For a future customer-facing version, it must say it's an AI, hand safety-critical questions to a person, and keep no chats without consent.
03What changed
- The prototype convinced the board to extend my contract so I could build the real thing.
- 100% correct answers on the 120-question test, against an 80% target.
- Rolled out to the 15-person customer support team.
- Answers in about a minute, never more than two, instead of 2 to 3 hours.
“Rather than approaching AI as a theoretical exercise, he focused on identifying where it could genuinely improve the way we worked... and then implemented tools and processes that delivered real efficiency gains.”
Before I built the tool, I decided what 'good enough to trust' meant.