My Role and Contribution
As a solo designer, I created the design from identifying the opportunities, prototyping and measuring the success. My contributions included:
- Wrote the growth experiment specifications and built the action plan with my team.
- Created different Hi-Fi prototypes for A/B tests.
- Measured the success and validated the outcome.
Timeline
6 Weeks
My Team
One iOS engineer lead and one growth engineer
Problem
The D1 unbound retention was low, and user engagement overall could have been better. How might we conduct a growth experiment to see significant results on improving D1 retention?
Step 1 Research
When users follow more than 1 person and repost more than 1 time, the Day 1 retention can reach approximately 40%. We believe that an increased conversion rate of follows and reposts incentivizes users to find value and spend more time in Firework, which drives an increase in day 1 retention.
Step 2 Outline Experiments
We believed the conversion rate would increase on the follow button after we made it solid red; the current follow button on the profile and notification page were easily ignored by users since the symbols were not obvious.
To verify this, we conducted A/B testing on the profile and notification page. There was a control group, with 50% of users viewing the existing layout of other people’s profiles and the notification page, and an experiment group, where 50% of users saw the red follow button on other people’s profiles and the notification page. We measured which group had a better conversion rate and used the A/B testing calculator to be certain of success.
The conversion rate for the follow button on the homepage was much higher than the one for the notification page. We believed there would be an increase in conversion rate on the follow button after we hided some CTAs and arranged the position of the CTAs because too many CTAs increased the cognitive overload which in our case drove the traffic from the follow button to other CTAs that were unrelated to day 1 retention. Additionally, the follow CTA was not grouped together with other important CTAs which isolated the traffic.
To verify this, we conducted A/B testing on the video page. There was a control group, where 33% of users saw the existing layout, and an experiment group, where 33% of users saw the follow button moving from top to bottom, and the collection icon was hidden. In the B experiment group, 33% of users saw the title plus the hashtag at the top, with the follow button moving from top to bottom, and the collection icon was hidden. We measured which group had a better conversion rate and used the A/B testing calculator to be certain of success.
We believed that there would be an increase in the conversion rate for the repost button after we explained the CTA well and added gesture to trigger the repost because users had trouble understanding the repost icon.
To verify this, we conducted A/B testing on the video feed page. There was a control group, where 33 percent of users saw the existing layout, and the A experiment group, where 33 percent of users saw a text label explaining the repost icon. In the B experiment group, 33 percent of users saw a text label for the repost icon as well as a double tap to repost. After twos on the third video, it showed a double tap to repost the tool tip to educate people about the gesture. The animation zoomed in the gesture. We measured which group had a better conversion rate and used the A/B testing calculator to be certain of success.
Step 3 Measure success
For users who enter the notification page, we saw a statistically significant 38 percent increase in follow conversion for the A experiment, and D1 retention increased from 45.3 percent to 49.7 percent when we displayed a follow button with a prominent color. High-converted CTA buttons attract attention and tell users what to do as soon as they land on a page.
For users who enter the video feed, there was a statistically significant 12 percent increase in follow conversion for the A experiment, and D1retention increased from 45.3 percent to 47.6 percent by making the CTA easy to reach. Research has shown that 75 percent of interactions are thumb-driven. It is important to ensure key CTAs are easy to reach and in the thumb zone for most users.
We saw a statistically significant increase of repost conversion both on IOS and Android for the B experiment, from a 5.59 percent conversion rate to a 14.0 percent conversion rate, and from a 7.13 percent conversion rate to a 14.5 percent conversion rate. Gestures are intuitive, and once a user learns how to use them, gestures become an important part of the mobile user experience.
Next Step and Reflection
We roll out successful experiments company-wide in a 2 week cycle. If I had more time with this project, I'd like to measure its success through qualitative research. Understanding how users reacted and felt matters as much as the data behind their behavior.
