Investigating a Gradual Drop in Conversion Rate: A Case Study for a Fashion E-commerce Giant- Part I
A Data-Driven Tale of How Analytics Saved the Day
It was a rainy evening in Bangalore, and Avantika, the Head of Product at an Indian e-commerce giant specialising in fashion and lifestyle products, was meeting with her analytics counterpart, Rohit. As they sipped on their hot cups of tea, Avantika voiced her concern about the gradual drop in session level conversion rate compared to last month. Rohit quickly pulled up the conversion dashboard and confirmed Avantika's observation. They both knew that a drop in conversion rate could significantly impact the company's bottom line, and it was crucial to investigate the root cause of the problem.
"Have you noticed any changes in the mix of our cohorts?" Rohit asked, looking over the data. "Perhaps the new users aren't converting as well as the old ones."
"That's a good point," Avantika replied, "But we haven't made any significant changes to our acquisition strategy recently."
"What about any recent changes to the product or the user experience?" Rohit asked, scrolling through the data.
"I don't think so," Avantika said, thinking hard. "But let's double-check."
Rohit nodded and promised to conduct a thorough analysis of the data to identify any potential issues.
Based on the initial conversation between Avantika and Rohit, they both had a few hypotheses about the possible cause of the drop in conversion rate. However, before jumping to any conclusions, they needed to conduct a thorough investigation of the data. Rohit's promise to conduct an analysis was just the beginning of a lengthy process that would require them to dive deep into the data and explore different variables.
Avantika and Rohit knew that this was not an easy task, but they were determined to get to the bottom of the issue. They realised that every aspect of their business, from the product and user experience to marketing and advertising, could potentially impact the conversion rate. Therefore, they needed to be meticulous in their approach and consider every possible angle.
Initial Analysis
Rohit knew that the issue could stem from various factors. He started by analysing the user experience of the app, but after going through the design and user interface, he found only minor issues that were unlikely to cause such a significant drop in conversion.
He then explored the pricing hypothesis but did not find any significant changes in pricing or discounts that could have led to the decline in conversion. He also investigated whether there was a change in the user mix, but found no significant differences in the demographic distribution of users during the period.
Additionally, Rohit ran various tests on the app's functionality to check if there were any technical issues that could have caused the drop in conversion rate, but found no glitches that could be responsible.
Even after thorough analysis, Rohit could not find any significant changes that could explain the decline in conversion rate. He was feeling increasingly frustrated and overwhelmed by the sheer volume of data that he had collected. He realised that without a clear structure in his analysis, he was getting lost in the sea of data and unable to make any meaningful conclusions.
Deconstructing the conversion rate: Cohort and Funnel Analysis
Rohit's experience highlights the importance of having a structured approach to data analysis. Without a clear structure, it's easy to get lost in the vast amount of data and lose sight of the goal. A structured approach helps to break down the problem into manageable pieces and allows for a more systematic investigation of potential causes.
Breaking down metrics into different dimensions is essential for a systematic and thorough investigation of potential causes for any issue or problem. This deconstruction allows for a more in-depth analysis of different aspects of the business, making it easier to identify specific areas that might be contributing to the problem.
Cohort and funnel analysis are two dimensions that are particularly useful in investigating the root causes of changes in conversion rate. Cohort analysis can be thought of as the vertical dimension as it enables businesses to understand how different user groups behave over time, allowing for the identification of specific user groups that might be contributing to the problem. Funnel analysis, on the other hand, can be seen as the horizontal dimension as it helps to identify which parts of the user journey might be causing issues, such as drop-offs in the user journey.
By breaking down the conversion rate into two dimensions, businesses can approach the problem in a more structured and targeted way. This can help to save time and resources that might otherwise be spent on investigating irrelevant factors or variables.
Moreover, by focusing on specific dimensions, businesses can develop more targeted and effective solutions to the problem. For instance, if the cohort analysis highlights that a specific user group is experiencing a lower conversion rate, businesses can develop specific strategies to address the issues faced by this group. Similarly, if the funnel analysis identifies that users are dropping off at a specific stage of the user journey, businesses can focus on improving that stage to reduce the drop-off rate.
Overall, breaking down metrics into different dimensions, such as cohort analysis and funnel analysis, is crucial for a systematic and thorough investigation of potential causes. This approach enables businesses to identify specific areas that might be contributing to the problem, develop targeted solutions, and save time and resources in the process.
Cohort Analysis
Rohit took a structured approach to analyse the conversion rate by deconstructing it into different user cohorts. He began by analysing cohorts based on user acquisition channels, including organic search, social media, and paid advertising. Rohit observed that the conversion rate had dropped across all channels, ruling out the possibility of acquisition issues.
Next, he looked into user type cohorts, such as new users and returning users. Rohit found that the drop in conversion rate was across new as well as returning users. But the quantum of drop was more pronounced among new users than returning users, which led him to investigate the onboarding process for new users and identify areas for improvement.
Further, Rohit analysed cohorts based on the device used by the users, such as desktop and mobile. He discovered that the conversion rate had dropped more on mobile devices than on desktops. This led him to investigate the mobile user experience and identify potential issues.
Moreover, Rohit analysed cohorts based on the user location, such as users from different regions or countries. He found that the conversion rate had dropped across all the regions.
Finally, Rohit analysed cohorts based on the product category, such as clothing, footwear, and accessories. He observed that the conversion rate had dropped more for certain product categories such as ethnic wear and personal care, leading him to investigate potential issues with those specific product categories.
Rohit's structured approach to analysing the conversion rate has provided him with a clear direction for further investigation. He has identified specific user groups that are contributing to the drop in conversion rate, such as new users, mobile users, and users of certain product categories. However, he still needs to pinpoint the exact source of trouble within these cohorts. This is where funnel analysis comes into play. By analysing the steps users take before completing a desired action, such as making a purchase, Rohit can identify where users are dropping off in the conversion funnel. This will help him identify specific issues that need to be addressed to improve the overall conversion rate.
Funnel Analysis
A funnel analysis is a method of analysing a user's journey through a website or app by breaking it down into a series of steps, or stages, which are then analysed to identify any points of drop off or conversion.
Rohit looked at the trend line of the drop between consecutive steps in the funnel. This helped him to identify the exact point in the user journey where the drop-off was occurring and develop strategies to address the issue.
To perform a funnel analysis, Rohit began by defining the different steps in the user journey on the app. He identified the following stages:
All Users: This is the total number of visitors to the app
Non-bounced Users: These are users who did not leave the app after opening it
Users with Product page view: These are users who clicked on a specific product to view its details
Users with Add to Cart: These are users who added a product to their cart
Users with Checkout: These are users who proceeded to the checkout process after adding a product to their cart
Users with Orders: These are users who completed a purchase on the app
By analysing the funnel, Rohit could see how many users were dropping off at each stage of the journey. He observed that there was a significant drop between non-bounced users and users with a product page view. This drop-off is a critical metric to analyse because it represents the point where users start to engage with the app's content.
To gain a deeper understanding of the drop between non-bounced users and users with a product page view, Rohit decided to break down the user journey between the two stages into three different paths: users who reach the product page through search, users who reach the product page through product recommendation, and users who reach the product page through category navigation.
Users who reach the product page through search: These are users who use a search engine or an internal search bar on the app to find a specific product. They enter relevant keywords or phrases into the search bar and click on the product that matches their search criteria.
Users who reach the product page through product recommendation: These are users who are directed to a specific product through a recommendation algorithm on the app. The algorithm may use factors such as the user's past purchase history, browsing history, or items in their cart to suggest products that may be of interest to them.
Users who reach the product page through category navigation: These are users who navigate through different categories and subcategories on the app to find the product they are interested in. They may use menus or filters to refine their search and eventually land on the product page.
After breaking down the user journey into these three paths, Rohit discovered that the drop-off was most prominent for users who reached the product page through search. This indicated that there may be issues with the app's search functionality or the relevance of search results.
Rohit decided to strengthen his hypothesis by overlaying the funnel analysis with user cohorts. His analysis revealed that the drop-off rate was notably higher for a specific group of users - new mobile users who were engaging with identified product categories. This information allowed Rohit to pinpoint the exact issue and concentrate his efforts on a particular cohort and segment of the user journey, leading to more targeted and effective solutions.
Rohit entered Avantika's office with a confident stride, holding a printout of the funnel analysis in his hand. He explained how he had broken down the user journey and identified the drop-off points, ultimately pinpointing the problem with the search algorithm for new mobile users engaging with specific product categories. Avantika was impressed with Rohit's findings and commended him on his thorough analysis. They both agreed to present the findings to the data science team responsible for the search algorithm and work on further deep dive.
In this first part of my blog, we explored how cohort analysis and funnel analysis can provide valuable insights into user behaviour on an app. Cohort analysis allows us to group users based on shared characteristics and observe how they behave over time, while funnel analysis helps us understand how users move through different stages of a conversion process.
We then saw how overlaying these two types of analyses can provide even deeper insights into user behaviour. By breaking down the user journey into different paths and overlaying the analysis by user cohorts, Rohit was able to pinpoint the problem areas in the app and focus his attention on improving the user experience for specific cohort and part of the user journey.
In the upcoming part of the blog, we will witness Rohit's efforts in concluding the analysis with definitive findings. We will observe how he handles a potential conflict between data science and analytics by utilizing the concept of causal inference. Along the way, we will learn about the significance of avoiding common pitfalls in analysis, such as selection bias, and the importance of conducting experiments to ensure accurate and actionable insights. Stay tuned for more exciting insights!