Speed Meets Structure: Rapid Testing and Metrics Framework Development for the Terms Chatbot

Research showed attorneys wouldn’t trust an AI answer they couldn’t verify, and designing around that helped more than double satisfaction with Intapp Assist for Terms.

Overview:

It's important for any attorney to prevent invoice rejections, ensure timely payments, and maintain strong client trust, avoiding lost revenue and continuing to build relationships with clients. The team knew from support tickets and the Client Success team that it was tedious for attorneys to pull up specific pieces of information from terms agreements.

Project Scope

  • Company; Intapp

  • Timeframe: 2 3-week sprints

  • My Role: UX Researcher

  • Teams Involved: Design, Engineering, Product Management

  • Methods: Usability Tests, Product Analytics

  • Tools: Figma, Microsoft Teams, Slack, Pendo

  • Constraint:

    • Participant access: Lawyers and legal support staff were the primary users, and their time was scarce. Recruiting meant working around heavy workloads and customer communication schedules.

Feature: Intapp Assist for Terms

Intapp Assist for Terms is a bot that pulled relevant term details depending on the question it was asked and was located inside Microsoft Teams, a platform we knew was already widely used by attorneys and their support staff. The goal of the bot was to help attorneys pull up specific pieces of information from terms agreements efficiently and effectively, streamlining a task that was otherwise time-consuming and manual.


Key Question:
How does this solution impact the way attorneys and support staff find and use terms information?


Business Goal:
Increase time to value and the likelihood of tool adoption and retention, opening up an additional revenue stream tied to Intapp AI.

User Goal:
Identify ways to improve legal professionals' experience looking up specific terms when billing clients and generating revenue for their firms.

Research Phases and Key Insights

Phase 1: Usability Testing During Early Access (Sprint 1)

Goal: Learn whether the chatbot helped users find specific terms faster than their current process.

n = 13

Key Takeaways:

  • What worked: Users found the chatbot easy to locate, the tone fit their needs, and the output was easy to scan.

  • What didn’t: Users couldn’t verify answers without source links, so they still scanned the terms agreement manually to confirm accuracy. The chatbot also handled misspellings inconsistently.

Phase 2: Usability Testing During Early Access (Sprint 2)

Goal: Test whether our changes fixed the issues from Sprint 1: verifying answers without source links and inconsistent handling of misspellings.

n = 13

Key Takeaways:

  • What improved: Users could verify answers through source links, and the chatbot handled misspellings more reliably.

  • What still needs work: Users expected source links in an expandable dropdown, not in their current format. Misspelling handling also improved but still fell short of what users expected.

  • What we learned next: Response time mattered. Users lost patience after 3–5 seconds unless they had some sense of how long the answer would take.

Phase 3: Metrics Framework Development

Goal: Give the team a way to measure the chatbot’s impact over time and decide where to invest next.

Key Metrics:

  • Adoption: How many users tried the chatbot

  • Retention: How many came back to use it again

  • Satisfaction: CSAT scores and thumbs up/down ratings on responses

I helped set up Pendo to track these metrics, and the team has since taken ownership of ongoing measurement.

Deliverables

  • Research findings decks with recommendations

  • CSAT dashboard in Pendo

Outcomes Tied Back to the Challenge

  • CSAT rose from 30% to 70%: Satisfaction more than doubled between the end of the Early Adopter program and three months after public launch.

  • 15+ usability improvements: Shipped fixes from two rounds of testing, addressing source verification and misspelling handling.

  • Laid the groundwork for a trust framework: Used what we learned about users’ need to verify answers to shape how the team approaches trust in AI output.

    What I’d do differently: Track trust and accuracy from the start with signals such as source-link clicks or thumbs-down reasons. Our metrics covered adoption, retention, and satisfaction, but Phase 1 showed trust was the core issue.

Previous
Previous

0-to-1 Research: Making the Case for Data Hygiene in AI Product Development

Next
Next

Testing Early AI Concepts to Close the SMS Adoption-Retention Gap