Testing Early AI Concepts to Close the SMS Adoption-Retention Gap
Users tried Klaviyo’s SMS Assistant once and didn’t come back. Research showed us why, and how to make an AI message generation tool brands actually want to use.
Overview:
One of the core challenges our users faced was composing SMS messages that would engage their customers and drive purchases. While Klaviyo had already built and shipped a message generator, early feedback signaled friction.
Rather than committing further engineering effort and budget without validation, we tested the experience directly with users to understand where it was falling short.
Project Scope
Company; Klaviyo
Timeframe: 5 weeks
My Role: UX Researcher
Teams Involved: Design, Engineering, Product Management
Methods: Usability Testing, Wizard-of-Oz, HaTS Survey, Customer Interviews
Tools: Figma, Zoom, Qualtrics, Slack
Constraints:
Research tied to engineering and go-to-market roadmaps
Testing a new AI model without engineering time we did not have
Feature: SMS Assistant
The goal of the assistant was to help Klaviyo users write personalized text messages that would generate engagement and sales for their brands.
Key Question:
How might we increase user engagement and introduce helpful technology that enables users to send messages through the SMS channel more efficiently and effectively?
Business Goal:
Identify ways to increase SMS channel adoption and retention through natural language capabilities.
User Goal:
Understand how the message assistant supports creative, personable message creation while addressing user pain points.
Research Phases and Key Insights
Why this mix: We wanted to understand the general level of satisfaction, how it tracked over time, and what was influencing those numbers along with identifying and mitigating any risks of the AI tool launch.
Phase 1: HaTS Survey
Goal: Establish a general baseline for user satisfaction and usability to measure future improvements against.
n = 120
Key Takeaways:
Average Satisfaction levels: The average satisfaction pre-improvements landed at 3.3 out of 5, with several tasks scoring notably lower. This signaled clear room for improvement, and recurring themes in the responses pointed to specific gaps between user expectations and the actual experience.
Retention dropoff: Shortly after these initial results, we found that retention dropped 50 points from initial adoption, an indication that users were trying the tool, finding it unsatisfying, and disengaging rather than returning.
Phase 2: Current Experience Walkthrough + Wizard of Oz Tests
Goal: Identify low-hanging friction points and understand how we can improve the proposed experience ahead of launch.
n = 7
Key Takeaways:
Positive signals: Users were open to engaging with NL-generated messages and curious to see what the tool would produce.
Concerns: Accuracy and the ability to easily tailor messages to their brand voice were the two biggest sticking points.
Value: Users appreciated the tool most as a thought partner in drafting messages.
Recommendations implemented from this research:
Give users greater control over tone before messages are generated, as well as over the final message.
Make it easier for users to locate where they can edit the final message.
Collect user context up front, which made the messages more tailored to what users were trying to achieve.
Deliverables
Research findings decks with recommendations
HaTS satisfaction dashboard
Cross-functional alignment workshop to prioritize next steps
Outcomes Tied Back to the Challenge
Smarter message generation: Changed the model to ask users for context before generating, so messages are tailored to each brand’s goals and voice.
10+ usability fixes: Shipped fixes from the walkthrough and Wizard of Oz sessions, including more control over tone and an easier-to-find editing step.
A baseline for measuring impact: Set a satisfaction baseline (3.3/5) with the HaTS survey so future improvements can be measured against it.
What I’d do differently: Pair Wizard of Oz testing with real model outputs as soon as they were available. Simulating the AI helped us learn quickly what users needed, but it couldn’t capture how they’d react to the model’s actual accuracy, which was one of their biggest concerns. Testing both together would have shown us earlier where the model fell short of users’ expectations.