This blog delves into the art and science of designing event tracking systems specifically for clickstream analytics in CRO. We’ll explore best practices for event nomenclature, granularity, standardisation, event properties, schema maintenance, and more. Whether you’re an analyst, product manager, data engineer, or analytics specialist, this guide will equip you with the knowledge and tools to build a robust event tracking framework that directly supports your conversion optimization goals.

By the end of this post, you’ll understand how to:

• Design events that capture meaningful user behaviors.

• Establish a scalable and maintainable event schema.

• Avoid common pitfalls in event instrumentation.

• Leverage event data to uncover actionable insights for CRO.

Let’s dive into the principles and practices that will help you create a high-quality event tracking system tailored for CRO success.

Event Nomenclature: Crafting Clear and Consistent Names

Event names are the foundation of clickstream analytics—they identify user actions clearly and consistently. A strong naming convention improves readability, ensures uniformity, and simplifies data analysis across teams and platforms.

Guidelines for Event Naming

• Be Descriptive Yet Concise: Use names that convey the action without excess words (e.g., button_clicked vs. user_clicked_the_submit_button).

• Follow a Consistent Structure: Use a verb-noun pattern (e.g., page_viewed, item_added, form_submitted) for uniformity.

• Use Snake_Case or CamelCase: Pick one format and stick to it (e.g., item_added or itemAdded) for readability and tool compatibility.

• Avoid Ambiguity: Skip vague terms like click or event; specify context (e.g., menu_expanded instead of click).

• Plan for Scalability: Add qualifiers to distinguish similar actions (e.g., checkout_started vs. checkout_completed).

Example Naming Conventions

• E-commerce: product_viewed, cart_updated, order_placed

• Content Platform: article_read, video_played, comment_posted

• SaaS Application: dashboard_loaded, report_exported, user_invited

Benefits

• Analysts interpret events quickly without heavy documentation.

• Engineers implement and maintain events with less confusion.

• Scalability improves as new features or products are introduced.

Granularity of Events: Striking the Right Balance

Event granularity defines the detail level in your tracking system. Too broad, and you miss insights; too detailed, and you’re buried in noise. The trick is aligning granularity with your business goals and analytical needs.

Determining Optimal Granularity

1. Start with Objectives: Identify what you need to measure (e.g., user engagement, conversion funnels, feature usage) and track supporting events.

2. Focus on Meaningful Actions: Capture significant interactions, not every micro-action (e.g., form_submitted over every keystroke).

3. Consider Downstream Impact: Ensure granularity supports reporting and analysis without overloading storage or processing.

Practical Advice

• Low Granularity (Broad): Use for high-level insights, like page_viewed for overall traffic.

• Medium Granularity (Balanced): Best for most cases, like button_clicked with properties like button_name.

• High Granularity (Detailed): Use for specific debugging or optimization, like scroll_depth_reached with thresholds.

Visual Aid: Granularity Flowchart

Here’s a flowchart to guide your granularity decisions:

Example Scenario

For an e-commerce checkout flow:

• Too Coarse: checkout_event (misses key steps).

• Too Granular: mouse_moved, field_focused (overloads pipelines).

• Just Right: checkout_started, payment_selected, order_placed (tracks the funnel effectively).

Event Standardisation and Taxonomy: Building a Unified Framework

A standardised event structure and taxonomy ensure consistency across products, platforms, and teams, making data interoperable and analysis-ready.

Best Practices for Standardisation

1. Define Categories: Group events into meaningful, logical categories that reflect user interactions clearly.

   

2. Create a Taxonomy: Establish a clear hierarchy for organising events:

   Event Taxonomy
   ├── Navigation
   │   ├── Viewed
   │   │   ├── Page (e.g., `page_viewed`)
   │   │   └── Menu (e.g., `menu_expanded`)
   ├── Interaction
   │   ├── Clicked
   │   │   ├── Button (e.g., `button_clicked`)
   │   └── Submitted
   │       └── Form (e.g., `form_submitted`)
   ├── Transaction
   │   ├── Started
   │   │   └── Checkout (e.g., `checkout_started`)
   │   └── Completed
   │       └── Order (e.g., `order_completed`)
   └── Engagement
       ├── Played
       │   └── Video (e.g., `video_played`)
       └── Reached
           └── Scroll Depth (e.g., `scroll_depth_reached`)

3. Document Everything: Maintain detailed documentation for each event, including:

   

Implementation Tips

• Use Prefixes for Multi-Product Setups: Clearly distinguish events across multiple products or environments (e.g., web_page_viewed, app_page_viewed).

• Standardise Property Names: Ensure properties within categories remain consistent for easier aggregation and analysis.
  

  

  
  

Event Properties: Adding Context to Actions

Event properties provide depth and critical context to raw event data, enabling richer analysis and actionable insights. Properly defined and structured event properties can significantly enhance the effectiveness of your analytics by providing precise details on user interactions, behaviors, and journeys.

Types of Event Properties

1. Event-Specific Properties

These properties describe the specific action taken by the user. They provide detailed context directly associated with each individual event.

2. User-Level Properties

User-level properties provide demographic, behavioural, or status information about the user performing the action, helping segment and analyse user cohorts.

3. Session-Level Properties

These properties describe the context of the session during which events occur. They help analyse user behaviour patterns within specific user sessions.

Guidance for Defining Event Properties

Keep Properties Relevant

• Only include properties that directly support business objectives or analytical needs.

• Avoid unnecessary properties that clutter data and complicate analysis.

Use Consistent Formats

• Standardize naming conventions for clarity (e.g., snake_case or camelCase).

• Ensure consistency in data types (dates in ISO format, monetary values as decimals).

Leverage Nested Structures

• Utilize JSON-like nested structures to represent complex data, especially when dealing with multiple related attributes.

Schema Cleanliness and Maintenance: Keeping Your Data Tidy

A clean event schema prevents redundancy, reduces errors, and ensures scalability.

Strategies for Schema Maintenance

1. Conduct Regular Audits: Quarterly checks for unused or duplicate events.

2. Deprecate Gracefully: Mark and phase out obsolete events.

3. Version Your Schema: Use versioning to update without disruptions.

4. Automate Validation: Implement checks to enforce standards during deployment.

Conclusion

Effective event design for clickstream analytics blends art and science. Clear naming, balanced granularity, standardized taxonomy, enriched properties, and schema maintenance unlock actionable insights and scalability. Start small, iterate, and collaborate to build a robust, evolving system that transforms clicks into valuable understanding.

Happy tracking!

Get Started: Sample Events and Template

Download our comprehensive Event Design Template to standardise and streamline your event instrumentation today.

To help you kickstart your event instrumentation, here are a few sample events to consider:

page_viewed:

{
  "event": "page_viewed",
  "properties": {
    "page_name": "product_detail",
    "page_path": "/products/wireless-headphones",
    "page_referrer": "/category/electronics",
    "page_title": "Wireless Bluetooth Headphones | Our Store",
    "scroll_depth": 0,
    "time_on_page_seconds": 0
  }
}

order_completed:

{
  "event": "order_completed",
  "properties": {
    "order_id": "ORD-12345",
    "currency": "USD",
    "total_value": 129.99,
    "tax_value": 10.40,
    "shipping_value": 5.99,
    "coupon_code": "SUMMER20",
    "discount_value": 26.00,
    "products": [
      {
        "product_id": "P-123",
        "product_name": "Wireless Headphones",
        "product_category": "Electronics",
        "product_price": 79.99,
        "product_quantity": 1
      },
      {
        "product_id": "P-456",
        "product_name": "Phone Charger",
        "product_category": "Accessories",
        "product_price": 19.99,
        "product_quantity": 2
      }
    ],
    "payment_method": "credit_card",
    "shipping_method": "standard"
  }
}

Google Tag Manager is a powerful, free tool from Google designed to simplify the process of managing and deploying marketing tags—snippets of code or tracking pixels used to collect data—on websites and mobile apps. It empowers marketers, analysts, and developers to efficiently implement and update tracking codes without the need for constant intervention from a web developer or direct modifications to the website’s source code. By using a container system, GTM allows all tags to be added, updated, and maintained in one centralized location, streamlining the process and reducing the risk of errors.

In essence, GTM acts as a bridge between your website and various marketing or analytics tools, enabling seamless data collection and improved workflow efficiency. Whether you're tracking user behavior, measuring campaign performance, or integrating third-party tools, GTM provides a user-friendly interface to manage it all with ease.

What is a Tag?

A tag is a small snippet of code—often JavaScript or HTML—that is added to a website or mobile app to collect data and send it to third-party tools. These tools can include analytics platforms (like Google Analytics), advertising networks (like Google Ads), or customer relationship management (CRM) systems. Tags are used to track various user interactions, such as page views, button clicks, form submissions, and e-commerce transactions. For example, a tag might be used to monitor how many users clicked on a specific call-to-action button or to measure the success of an online advertising campaign. While tags are essential for gathering insights and optimizing marketing efforts, managing them manually can be complex and time-consuming. This is where Google Tag Manager (GTM) comes in, simplifying the process of deploying and managing tags without requiring extensive technical expertise.

A Brief History of Tag Management

In the early days of digital marketing, tracking user behavior and collecting data on websites was a cumbersome, technically demanding, and often inefficient process. Before the advent of Tag Management Systems (TMS), businesses had to rely on manually embedding tracking codes—such as those for web analytics, advertising, conversion tracking, and remarketing—directly into their website’s source code. This traditional approach posed several challenges:

• Heavy Developer Dependency: Every time a new tag needed to be added, updated, or removed, developers had to modify the website’s code manually. This process was not only time-consuming but also led to operational bottlenecks, delaying marketing campaigns and data collection.

• Error-Prone Implementation: A misplaced snippet, syntax error, or incorrectly deployed tag could break website functionality or lead to inaccurate data collection, making reliable analytics difficult.

• Performance Issues: As websites accumulated multiple tracking tags, page load speeds suffered. Excessive scripts could slow down website performance, negatively affecting user experience and even impacting SEO rankings.

As digital marketing evolved, businesses became more data-driven, relying heavily on real-time insights to optimize their strategies. However, the limitations of manual tag implementation became more apparent. Marketers needed the ability to deploy and manage an increasing number of tags to track critical user interactions.

Without a centralized system to handle these diverse tracking needs, businesses faced inefficiencies, inconsistent data collection, and a lack of agility in responding to new marketing demands. The increasing complexity of digital marketing necessitated a more scalable, efficient, and user-friendly solution—paving the way for Google Tag Manager (GTM) and other Tag Management Systems.

The Rise of Tag Management Systems

The tag management revolution gained momentum between 2008 and 2011, as businesses struggled with the inefficiencies of manually managing tracking codes. Recognizing this challenge, pioneering solutions like Satellite (later acquired by Adobe and integrated into Adobe Launch) emerged as some of the first dedicated Tag Management Systems (TMS). Soon after, companies like Tealium and Ensighten entered the market, further solidifying the importance of a streamlined approach to tag deployment. These early innovations demonstrated the immense potential of tag management for marketing and analytics teams by enabling them to:

• Deploy Tags Independently: Reduce reliance on developers, allowing marketers to implement tracking codes quickly and efficiently without modifying website code.

• Centralize Tag Management: Provide a single interface to add, update, and organize all tags, ensuring consistency and simplifying maintenance.

• Improve Efficiency and Accuracy: Minimize human errors, reduce broken or outdated scripts, and enhance data reliability for better decision-making.

As businesses increasingly relied on data-driven marketing, they began to recognize the strategic advantage of a centralized tag management solution. By shifting away from manual implementations, organizations not only improved operational efficiency but also enhanced website performance—reducing unnecessary code bloat and ensuring faster load times. This shift set the stage for the next generation of tag management tools, with Google Tag Manager (GTM) emerging as a game-changer in the industry.

The Emergence of Google Tag Manager

As businesses recognized the need for an efficient and scalable tag management solution, Google disrupted the market in 2012 by launching Google Tag Manager (GTM)—a free, user-friendly, and powerful tag management system that quickly became the industry standard. Unlike earlier paid solutions that required significant technical expertise, GTM democratized tag management, making it accessible to marketers, analysts, and developers alike.

What set GTM apart was its:

• Intuitive Interface – A no-code environment that allowed marketers to manage tracking codes without extensive developer involvement.

• Seamless Integration with Google Ecosystem – Native support for Google Analytics, Google Ads, and other Google tools.

• Flexible Triggers and Variables – A highly customizable system to track user interactions dynamically.

• Version Control and Debugging Tools – Features like built-in Preview Mode and change history helped teams test and troubleshoot before deployment.

• Scalability – Whether for small businesses or large enterprises, GTM provided a centralized platform for managing tags across websites and mobile apps.

By offering a free, robust, and scalable tag management system, Google Tag Manager quickly became the go-to choice for businesses looking to streamline their tracking efforts, improve website performance, and gain deeper insights into user behavior.

The Philosophy of GTM: Breaking It Down

At its core, Google Tag Manager (GTM) is a system that helps you communicate with various tracking tools (like Google Analytics, Google Ads, Facebook Pixel, etc.) by managing and deploying tags on your website or app. But rather than diving straight into the technical details, let’s simplify it with a real-world analogy.

Think of GTM Like a Football (Soccer) Match

Imagine you’re watching a football (soccer) match. The striker’s job is to score goals, but to do this successfully, three things need to align:

1. The Action – The striker kicks the ball toward the goal to score.

2. When the Action Happens – The striker decides to shoot when they’re 10 meters away from the goal.

3. Additional Information – The striker considers factors like wind direction, goalkeeper’s position, and shot angle to ensure a successful goal.

This framework—Action, When, and Additional Information—is the foundation of how GTM operates when tracking user interactions.

Applying the Framework to GTM

Let’s say you have a website button labeled “Click”, and you want to track when users click it. Using the same football-inspired framework, here’s how GTM helps:

1. The Action → Capture the button click and send it to Google Analytics as an event.

2. When the Action Happens → The event should trigger when a user clicks the “Click” button.

3. Additional Information → Ensure that you’re only tracking this specific button and not others (e.g., a second button labeled “Click Two”).

By thinking in this structured way, GTM becomes more intuitive and manageable—every task can be broken down into these three simple components.

Breaking Down GTM’s Three Core Components

At the core of Google Tag Manager (GTM) are three essential components: Tags, Triggers, and Variables. These elements work together to track user interactions, send data to analytics and marketing platforms, and automate tracking tasks without modifying website code. Understanding how these components interact is key to efficiently using GTM.

Tags: “What Needs to Be Done?”

A tag is an instruction for Google Tag Manager (GTM) to carry out a specific task, such as adding or running code on your website. Tags enable you to collect data, track user behavior, and integrate with third-party tools. Common examples include:

• Sending data to analytics platforms like Google Analytics.

• Informing advertising networks (e.g., Google Ads, Meta Ads) about conversions or user behavior.

• Running custom HTML or JavaScript code to modify website elements.

Think of tags as commands that tell GTM what to do. Each tag specifies:

1. The Action: What task should be performed (e.g., track a conversion or load a pixel).

2. Execution Conditions: When the tag should run (determined by triggers).

3. Dynamic Data: Additional details (provided by variables) to make the action context-aware.

Tags don’t execute automatically—they require triggers to activate them.

Triggers: “When Does It Happen?”

A trigger determines when a tag should fire based on specific conditions. Triggers act as gatekeepers, ensuring tags only execute at the right time. Common trigger types include:

• Page Loads: Firing a tag when a specific page is viewed.

• Button Clicks: Activating a tag when a user clicks a button.

• Form Submissions: Triggering a tag when a form is submitted.

Key Features of Triggers:

• Event-Based Activation: Triggers respond to user or system events, like a button click or page view.

• Condition Checking: GTM evaluates real-time data (e.g., page URL, click text) to determine if the conditions are met.

• Flexibility: A single tag can have multiple triggers, and a single trigger can fire multiple tags. For example, a purchase confirmation trigger might fire both a Google Analytics event tag and a Google Ads conversion tag.

Triggers ensure your tags execute precisely when and where they’re needed.

Variables: “What Extra Details Are Needed?”

A variable is a placeholder that stores or retrieves specific information, such as a page URL, product price, or user action. Variables make tags and triggers dynamic by providing the necessary details to:

• Identify Context: Which page is the user on? Which button did they click?

• Pass Data to Tags: What is the purchase amount? What is the product name?

Types of Variables:

1. Built-In Variables: Pre-configured options like Page URL, Click Text, and Click ID.

2. User-Defined Variables: Custom variables for specific needs, such as Order Total or Button Color.

Variables ensure your tags and triggers are context-aware and accurate.

How They Work Together

To see how tags, triggers, and variables interact, let’s walk through a common scenario:

Example: Tracking a Purchase Confirmation

1. The Tag (The Action):

   • A Google Ads Conversion Tag is created to track purchases.

   • The tag says, “Send the purchase amount to Google Ads when a conversion is detected.”

2. The Trigger (The Condition):

   • The trigger is set to fire when the user lands on the “Thank You” page.

   • GTM evaluates the Page URL variable to confirm the user is on the correct page.

3. The Variables (The Extra Data):

   • A variable called Order Total captures the purchase amount from the data layer.

   • The trigger checks: “Does the Page URL contain ‘/thank-you/’? If yes, fire the Google Ads tag.”

   • The tag includes the Order Total variable, sending the purchase amount to Google Ads.

This seamless interaction ensures accurate and efficient data collection.

The Data Flow Within GTM’s Architecture

Understanding how data moves within Google Tag Manager (GTM) is essential for optimizing tracking accuracy and ensuring seamless communication between your website and external analytics or marketing platforms. GTM operates through a structured data flow that enables efficient data collection, processing, and transmission.

Step-by-Step Breakdown of GTM’s Data Flow

1. User Interaction (Event Occurs)

   • A user performs an action on the website, such as clicking a button, submitting a form, or making a purchase.

   • This interaction generates event data that GTM can process.

2. Trigger Evaluation

   • GTM checks whether any triggers are set up to detect this specific user action.

   • If a trigger condition is met (e.g., the user lands on a "Thank You" page after checkout), it activates the corresponding tags.

3. Variable Resolution

   • If the tag requires additional data (e.g., transaction amount, product ID, or user information), GTM fetches the relevant variables from either the built-in options, user-defined variables, or the data layer.

   • The data layer acts as a structured repository that temporarily holds event-specific information before GTM processes it.

4. Tag Execution

   • Once the trigger fires and variables are retrieved, GTM executes the tag.

   • This could mean sending event data to Google Analytics, firing a conversion tag for Google Ads, or passing data to a third-party tool like Facebook Pixel.

5. Data Transmission to External Platforms

   • The processed data is sent to the designated analytics or marketing platform (e.g., Google Analytics, Google Ads, Meta Ads).

   • These platforms then use the data for reporting, attribution, and optimization.

How the Data Layer Enhances GTM’s Data Flow

The data layer plays a crucial role in GTM’s architecture by acting as an intermediary between the website and GTM. It allows structured storage and retrieval of key user actions, reducing reliance on direct page elements and improving tracking flexibility.

Example Use Case:

• Instead of directly pulling an order value from the page (which might be unreliable due to dynamic content changes), the data layer stores the order amount when a user completes a purchase.

• GTM then retrieves this value dynamically using a variable and includes it in the conversion tag.

Visualizing the Data Flow in GTM

By following this structured approach, GTM ensures that tracking remains scalable, accurate, and efficient, reducing the need for hard-coded tracking scripts and enabling businesses to adapt their tracking strategy easily.

Common GTM Use Cases: How Businesses Leverage It

Google Tag Manager (GTM) is more than just a tool for simplifying tracking—it’s a critical component of data-driven decision-making for businesses of all sizes. From small startups optimizing their digital marketing to Fortune 500 companies running complex multi-channel analytics, GTM plays a pivotal role in streamlining tracking, improving campaign performance, and enhancing user experience.

Let’s explore how businesses—both small and large—leverage GTM across different industries, with real-world case studies and success stories.

1. E-commerce: Driving Sales and Optimizing Conversions

How GTM Helps:

• Tracks product views, cart additions, and completed purchases.

• Fires conversion tags for advertising platforms (Google Ads, Meta Ads, TikTok Ads).

• Helps in Dynamic Remarketing by sending product data to ad platforms.

• Tracks checkout funnel abandonment for CRO (Conversion Rate Optimization).

Case Study: ASOS – Scaling Global E-commerce Tracking

Challenge: ASOS, a global online fashion retailer, needed a scalable tracking system across multiple countries and currencies while keeping load times minimal.

GTM Solution:

• Implemented enhanced e-commerce tracking via Google Analytics through GTM.

• Created dynamic triggers to fire event tags only when users completed key actions.

• Used server-side tagging to improve data accuracy and comply with privacy regulations.

Impact:

• 25% increase in ad efficiency by sending precise conversion data to Google and Facebook Ads.

• Improved real-time tracking of global user behavior across 196 countries.

How Small E-commerce Stores Use GTM

Even smaller online businesses benefit from GTM by:

• Tracking which marketing channels drive the most sales.

• Setting up multi-touch attribution to understand the customer journey.

• A/B testing checkout flow changes to improve conversions.

2. Lead Generation: Capturing High-Quality Leads

How GTM Helps:

• Tracks form submissions, phone calls, and chat interactions.

• Fires conversion pixels for Google Ads and LinkedIn Ads.

• Measures lead quality by integrating with CRM tools like HubSpot or Salesforce.

Case Study: HubSpot – Streamlining B2B Lead Tracking

Challenge: HubSpot, a leading B2B marketing software company, struggled with inconsistent lead attribution across its paid and organic channels.

GTM Solution:

• Implemented form submission tracking to capture leads with precise UTMs.

• Used custom JavaScript tags to send CRM-enriched data to Google Ads.

• Set up triggers to differentiate between high-intent leads (e.g., demo requests) and low-intent leads (e.g., newsletter signups).

Impact:

• 40% improvement in lead attribution accuracy across paid channels.

• Increased conversion rates by optimizing ad retargeting campaigns using first-party data.

How Small Businesses Use GTM for Lead Generation

• Local service providers track phone calls and form submissions to measure ROI.

• Digital agencies use GTM to integrate Facebook Pixel and Google Ads for precise conversion tracking.

• Law firms and real estate agents set up scroll-depth tracking to analyze which pages generate the most engagement.

3. Content Marketing & Engagement: Tracking User Behavior

How GTM Helps:

• Measures scroll depth, video plays, and outbound link clicks.

• Tracks engagement with interactive elements like quizzes, calculators, and PDFs.

• Helps publishers and bloggers optimize time on site and bounce rates.

Case Study: The New York Times – Data-Driven Journalism

Challenge: The New York Times wanted deeper insights into reader engagement to personalize content recommendations and increase subscriptions.

GTM Solution:

• Implemented scroll tracking to measure which articles users fully read.

• Used event tracking for podcast listens, newsletter signups, and video interactions.

• Sent real-time engagement data to a custom machine-learning model to recommend similar articles.

Impact:

• 20% increase in reader retention by serving personalized article recommendations.

• Improved ad revenue by optimizing content layout based on engagement metrics.

How Smaller Publishers & Bloggers Use GTM

• Bloggers track which articles drive the most newsletter signups.

• Digital media sites use heatmap integration with GTM to improve UI/UX.

• Marketers track outbound clicks to measure content referrals to affiliate partners.

4. User Experience & CRO: Improving Website Performance

How GTM Helps:

• Collects data for heatmaps, session recordings, and A/B testing tools.

• Helps teams test new landing pages and UI changes without developer involvement.

• Measures interaction with site elements like dropdowns, modals, and navigation menus.

Case Study: Airbnb – Using GTM for Experimentation & CRO

Challenge: Airbnb needed a fast way to test new checkout flows, pricing structures, and user interface elements without relying on developers.

GTM Solution:

• Used event tracking to measure button clicks, time spent on booking pages, and user drop-off points.

• Integrated with A/B testing tools like Google Optimize to experiment with pricing models.

• Tracked real-time changes to identify UX friction points.

Impact:

• 15% increase in completed bookings after optimizing checkout flow.

• Faster experimentation cycles, allowing the team to test UX changes in days instead of weeks.

How Startups & Small Businesses Use GTM for UX Improvements

• Early-stage SaaS startups track user onboarding progress using GTM.

• E-commerce brands use GTM to run heatmap & session recording experiments.

• Landing page optimizers track which CTAs perform best through click tracking.

5. Privacy & Compliance: Navigating GDPR & CCPA Regulations

How GTM Helps:

• Helps businesses implement cookie consent banners and tracking opt-outs.

• Enables server-side tagging to comply with GDPR/CCPA while maintaining data accuracy.

• Controls which user data is collected and sent to external platforms.

Case Study: European E-commerce Brand – GDPR Compliance with GTM

Challenge: A European e-commerce company needed to balance GDPR compliance with effective ad tracking without violating user privacy.

GTM Solution:

• Implemented Google Consent Mode via GTM to adjust tracking based on user consent.

• Configured server-side GTM to anonymize IP addresses before sending data to analytics tools.

• Used trigger blocking rules to prevent tags from firing for users who opt out of tracking.

Impact:

• Maintained 80% of ad conversion tracking while staying compliant with GDPR.

• Improved trust and transparency by giving users full control over their data preferences.

How Small Businesses Handle Privacy with GTM

• WordPress sites use GTM to integrate cookie consent banners.

• Healthcare websites configure GTM to restrict data collection on sensitive pages.

• Ad-driven businesses use server-side GTM to track conversions while complying with privacy laws.

Final Thoughts

GTM is an essential tool across industries, helping businesses of all sizes track user behavior, optimize conversions, and comply with privacy regulations. Whether it’s a global e-commerce retailer refining its ad tracking or a small SaaS startup improving lead generation, GTM provides a scalable, flexible, and privacy-compliant solution for modern digital analytics.

By leveraging real-time tracking, automation, and integration with marketing platforms, companies can make data-driven decisions that directly impact their revenue and user experience.

Why Tracking Matters

Every interaction—be it a page view, button click, or product purchase—creates data that can enhance user experiences, inform business strategies, and improve product performance. By leveraging robust tracking methods, organizations gain reliable insights into:

• User Engagement: Which pages do users spend the most time on?

• Conversion Funnels: Where do users drop off when completing a purchase or signing up?

• Content Effectiveness: What types of content drive the most engagement or revenue?

In his book Data-Driven Marketing, Mark Jeffery emphasizes the importance of accurate data collection, stating that "insights gleaned from poor data can be more damaging than having no data at all." This insight underscores the criticality of methodical and trustworthy tracking, whether performed on the client side or the server side.

The key question lies in how these interactions are tracked, and that often depends on whether the tracking happens on the client side or the server side.

Client-Side Tracking: Traditional Yet Ubiquitous

Overview

Client-side tracking involves embedding scripts within a webpage or mobile app that execute in the user’s browser. Platforms like Google Analytics, Facebook Pixel, and various marketing automation tools heavily rely on this approach.

How It Works

1. JavaScript Snippets: Small chunks of code (tags or pixels) are placed in the webpage’s source.

2. Event Capture: As users interact with elements on the page (clicks, form submissions, scrolls), the scripts log these actions.

3. Data Transmission: Tracked events are sent to third-party analytics servers, often in real time, where they are processed and displayed.

Strengths

• Ease of Deployment: Marketing and analytics teams can add or modify tracking with minimal reliance on developers.

• Real-Time Insights: Events are processed immediately, enabling agile campaign decisions or rapid testing.

• Rich Tooling: A wide range of analytics and ad-tech vendors offer easy-to-integrate client-side solutions.

Weaknesses

• Ad Blockers: Users increasingly rely on ad blockers or script blockers, potentially preventing these scripts from running altogether.

• Performance Impacts: Each additional script can delay page load times and lead to suboptimal user experiences.

• Data Vulnerability: Client-side data transfer can be intercepted or manipulated, raising concerns about accuracy and privacy compliance.

Case Study: A popular news portal found that 20% of its readership used ad blockers, leading to incomplete audience analytics data. As a result, the team struggled to accurately measure campaign ROI for premium content.

Server-Side Tracking: Modern and Secure

Overview

Server-side tracking shifts the data collection responsibilities from the user’s browser to your backend infrastructure. This approach focuses on logging events when they hit your server, rather than relying on browser-executed code.

How It Works

1. Backend Integration: Your application server (or a dedicated tracking server) captures relevant user actions (e.g., sign-ups, purchases) before rendering a response or after an API call.

2. Data Aggregation: Events can be aggregated, transformed, or validated within the server.

3. Forwarding and Storage: Processed data is then forwarded to analytics platforms or stored in a data warehouse, where it undergoes further analysis.

Strengths

• Resilience to Blockers: Server-side events aren’t impacted by client-based blockers or script restrictions.

• Data Integrity: Controlling how data is collected and forwarded (on your own server) reduces the risk of manipulation and data loss.

• Enhanced Security: Minimizes exposure of sensitive information in the user’s browser, improving compliance with regulations like GDPR or CCPA.

Weaknesses

• Higher Complexity: Requires deeper backend changes and can introduce architectural overhead when you have multiple data sinks.

• Latency Considerations: Round-trip communications to external analytics services can introduce slight processing delays.

• Infrastructure Costs: More server load and data handling can increase hosting expenses and maintenance efforts.

Quote: In Lean Analytics, Alistair Croll and Benjamin Yoskovitz suggest that "Good data offers clarity; great data drives action." Server-side tracking often provides more reliable data, but it can demand significant resources to implement and maintain.

Side-by-Side Comparison

Challenges and Caveats

1. Compliance Overheads: Regulations like GDPR require robust data handling. Server-side tracking can simplify compliance but demands rigorous policy enforcement.

2. Tool Compatibility: Not all third-party analytics tools provide straightforward server-side integrations. Custom solutions may be needed.

3. Ongoing Maintenance: Both approaches need consistent updates to remain accurate—browsers evolve, server architecture changes, and new privacy regulations emerge.

Real-World Use Cases

• eCommerce: A high-traffic online store (e.g., Amazon) needing accurate cart abandonment data may lean on server-side tracking to avoid data gaps caused by ad blockers. This ensures more reliable measurement of user pathways.

• Content Marketing: A blog or news portal might benefit from the instant feedback loop of client-side tracking, enabling quick content optimizations. For instance, BuzzFeed uses real-time user metrics to decide which headlines to push on social media.

• Enterprise SaaS: Companies handling sensitive customer information (like Salesforce) often prioritize server-side tracking to maintain stricter compliance and better data governance.

Hybrid Approaches: The Best of Both Worlds?

Many organizations adopt a blended strategy:

• Client-Side for Engagement: Real-time front-end metrics (page views, clicks, scroll depth) for rapid iteration on UX and campaigns.

• Server-Side for Mission-Critical Data: Securely tracking logins, transactions, or subscription details on the server to ensure reliable and compliant data collection.

This hybrid setup can deliver immediate insights while safeguarding crucial user data. Netflix, for example, collects client-side metrics to personalize user interfaces while relying heavily on server-side data to understand streaming quality and user retention rates.

Final Thoughts

Tracking is a cornerstone of modern digital strategy, but the choice between server-side and client-side methods can significantly affect data accuracy, performance, and compliance. While client-side tracking remains popular for its simplicity and immediacy, server-side tracking provides a powerful way to ensure data reliability and protect user privacy.

By understanding the nuanced strengths and weaknesses of both approaches, you can craft a tracking architecture that meets your business objectives, scales with your growth, and respects evolving privacy standards. In the words of Thomas H. Davenport and Jeanne G. Harris from Competing on Analytics: "You can’t manage what you don’t measure—accurate measurement is the bedrock of effective analytics." The key is to remain agile—ready to adapt as technologies and regulations shift, ensuring that your analytics strategy remains both effective and responsible.

References: ASA statement on p-values; Microsoft’s sample-ratio mismatch guidance; Microsoft on repeated testing during experiments.

Introduction

In today's data-driven world, making informed decisions about product changes is crucial. A/B testing offers a powerful framework for gathering actionable insights and making evidence-based decisions, particularly in web and app development. In this blog post, we delve into the essential principles of designing, executing, and analyzing an A/B test using a practical, real-world example.

The Experimental Context: A Widget Store's Coupon Code Dilemma

Our example centers on a fictional online commerce platform specializing in selling widgets. In an effort to boost sales, the marketing team proposes a strategy involving promotional emails that include a coupon code for discounts on widgets. This proposal marks a potential shift in the company's business model, as they have never offered coupon codes before. The team is cautious, however, due to findings from prior studies—such as Dr. Footcare's revenue loss following the introduction of coupon codes (Kohavi, Longbottom et al., 2009) and evidence from GoodUI.org suggesting that removing coupon codes can be beneficial (Linowski, 2018). These insights raise concerns that simply adding a coupon code field to the checkout page might negatively impact revenue, even if no valid codes are available. Users might become distracted searching for codes or even abandon their purchases altogether.

To assess these potential effects, we adopt a “painted door” approach. This involves implementing a superficial change—adding a non-functional coupon code field to the checkout page. When users input any code, the system responds with “Invalid Coupon Code.” By focusing on this minimal implementation, we aim to evaluate the psychological and behavioral impact of merely displaying a coupon code field.

Given the simplicity of this change, we’ll test two distinct UI designs to compare their effectiveness. Testing multiple treatments alongside a control allows us to discern not only whether the idea of adding a coupon code is viable, but also which specific implementation is most effective. This targeted A/B test is a crucial step toward determining the feasibility of adopting coupon codes as part of the company’s broader business strategy.

Subscribe now

Hypothesis

Adding a coupon code field to the checkout page will degrade revenue-per-user for users who start the purchase process.

To test this hypothesis, we consider two UI implementations. This A/B test is a critical step in assessing the feasibility of this business model.

Goal Metrics

The primary metric, or Overall Evaluation Criterion (OEC), is revenue-per-user, normalized to account for sample size variability. To measure the impact of the change, we define success metrics carefully. While revenue is an obvious choice, using the total revenue sum is not recommended due to sample size variations across variants. Instead, revenue-per-user provides a more accurate and normalized metric. For this metric, it is critical to determine the denominator:

1. Including all site visitors introduces significant noise, as a large portion of users never initiate checkout. These users do not interact with the modified checkout process and, therefore, cannot provide meaningful data on the impact of the change. Including them dilutes the results, making it harder to detect any real differences caused by the experiment.

2. Restricting to only users who complete the purchase process presents a skewed perspective, as it inherently assumes that the modification influences purchase amounts without considering its effect on conversion rates. This approach excludes users who might abandon the process due to the change, potentially missing a critical aspect of the experiment's impact.

3. The best choice is users who start the purchase process, as they are directly exposed to the change at the checkout stage. This refined approach improves test sensitivity by ensuring that only the users who interact with the modified UI are analyzed. By excluding unaffected users, such as those who browse without adding items to the cart or initiating a purchase, the metric becomes more precise. This specificity helps isolate the true impact of the coupon code field, avoiding noise introduced by broader user behaviors that are irrelevant to the experiment.

With this setup, our refined hypothesis becomes: “Adding a coupon code field to the checkout page will degrade revenue-per-user for users who start the purchase process.”

Hypothesis Testing

Before we can design, run, or analyze our experiment, let us go over a few foundational concepts relating to statistical hypothesis testing. First, we characterize the metric by understanding the baseline mean value—the average of the metric under normal, unaltered conditions—and the standard error of the mean. The standard error provides insight into the variability of our metric estimates and is crucial for determining the required sample size to detect meaningful differences. By accurately estimating this variability, we can size our experiment properly and assess statistical significance during analysis.

For most metrics, we measure the mean, but alternative summary statistics, such as medians or percentiles, may be more appropriate in specific contexts, such as highly skewed data distributions. Sensitivity, or the ability to detect statistically significant differences, improves when the standard error of the mean is reduced. This can be achieved by either increasing the traffic allocated to the experimental variants or running the experiment for an extended period. However, running longer experiments may yield diminishing returns after a few weeks due to sub-linear growth in unique users (caused by repeat visitors) and potential increases in variance for certain metrics over time.

To evaluate the impact of the experiment, we analyze revenue-per-user estimates from the Control and Treatment samples by computing the p-value for their difference. The p-value represents the probability of observing the measured difference, or a more extreme one, under the assumption that the Null hypothesis—that there is no true difference—is correct. A sufficiently small p-value allows us to reject the Null hypothesis and infer that the observed effect is statistically significant. But what constitutes a small enough p-value?

Typically, the scientific benchmark is a p-value less than 0.05. This threshold means there is less than a 5% probability of incorrectly concluding there is an effect when none actually exists, providing confidence in 95 out of 100 cases. Furthermore, another approach to determine significance is through confidence intervals. A 95% confidence interval defines a range where the true difference between Treatment and Control lies 95% of the time. If this interval does not include zero, it reinforces the conclusion that the effect is statistically significant. These tools collectively help establish the robustness of experimental findings, ensuring decisions are data-driven and reliable.

Statistical Power

Statistical power measures the ability of an experiment to detect a meaningful difference between variants when such a difference truly exists. In simple terms, it is the probability of correctly rejecting the null hypothesis when there is an actual effect. For instance, if a retailer is testing a new homepage layout, statistical power ensures that subtle but real increases in sales do not go unnoticed.

To achieve reliable results, experiments are often designed with 80-90% power, meaning there is a high likelihood of detecting true changes. Power is influenced by factors such as sample size and effect size; larger sample sizes tend to improve power, but overly small differences might still evade detection. For example, while a large e-commerce platform like Amazon might be interested in detecting a 0.2% increase in revenue-per-user due to its massive scale, a smaller startup might focus only on changes exceeding 5-10% because such increases are critical for their growth.

While statistical significance helps us understand whether an observed difference is likely due to chance, it does not always translate into practical significance. Practical significance asks a more business-oriented question: Is the observed change large enough to matter? For example, a 0.2% increase in revenue-per-user might be meaningful for billion-dollar platforms like Google or Bing, but for a small startup seeking rapid growth, a 2% increase might still fall short of expectations. Setting clear business thresholds for what constitutes a meaningful change is essential. For our hypothetical widget store, we define practical significance as a 1% or larger increase in revenue-per-user, recognising this as the minimum impact needed to justify potential costs or risks of implementation.

Designing the Experiment

We are now ready to design our experiment. We have a hypothesis, a practical significance boundary, and we have characterised our metric. We will use this set of decisions to finalize the design:

1. What is the randomisation unit? The randomisation unit for this experiment is the user.

2. What population of randomisation units do we want to target? We will target all users and analyse those who visit the checkout page. Targeting a specific population allows for more focused results. For instance, if the new text in a feature is only available in certain languages, you would target users with those specific interface locales. Similarly, attributes such as geographic region, platform, and device type can guide targeting.

3. How large does our experiment need to be? The experiment size directly impacts the precision of results. To detect a 1% change in revenue-per-user with 80% power, we will conduct a power analysis to determine the sample size. The following considerations influence size:

   • Using a binary metric like purchase indicator (yes/no) instead of revenue-per-user can reduce variability, allowing for a smaller sample size.

   • Increasing the practical significance threshold—for example, detecting only changes larger than 1%—can also reduce sample size requirements.

   • Lowering the p-value threshold, such as from 0.05 to 0.01, increases sample size needs.

4. How long should the experiment run? To ensure robust results, we will run the experiment for at least one week to capture weekly cycles and account for day-of-week effects. External factors like seasonality and primacy or novelty effects are also important:

   • User behavior can vary during holidays or promotional periods, affecting external validity.

   • Novelty effects (e.g., initial enthusiasm for a new feature) and adoption effects (e.g., gradual user adoption) may impact results over time.

Final Experiment Design

1. Randomization Unit: User

2. Target Population: All users visiting the checkout page

3. Experiment Size: Determined via power analysis to achieve 80% power for detecting a 1% change

4. Experiment Duration: Minimum of one week to capture weekly cycles and extended if novelty or primacy effects are detected

5. Traffic Split: 34/33/33% for Control, Treatment 1, and Treatment 2

By carefully designing the experiment with these considerations, we can ensure that the results are both statistically and practically significant. Overpowering an experiment is often beneficial, as it allows for detailed segment analysis (e.g., by geographic region or platform). This approach not only improves the robustness of results but also helps businesses uncover nuanced insights. For example, identifying specific trends in user behavior across different demographics can inform future product iterations and marketing strategies.

Furthermore, running a well-structured A/B test fosters a culture of data-driven decision-making within the organization. By investing in comprehensive experiment design and analysis, businesses can mitigate risks, allocate resources effectively, and achieve sustainable growth. Ultimately, the insights derived from this experiment will not only validate the feasibility of introducing coupon codes but also set a benchmark for future experimentation, reinforcing the importance of innovation and customer-centric strategies in a competitive market.

Running the Experiment and Getting Data

Now let us run the experiment and gather the necessary data. To run an experiment, we need both:

• Instrumentation to get logs data on how users are interacting with your site and which experiments those interactions belong to.

• Infrastructure to be able to run an experiment, ranging from experiment configuration to variant assignment.

Interpreting the Results

The data collected from the experiment is the foundation for actionable insights, but ensuring its reliability is critical. Before diving into the revenue-per-user analysis, it is essential to validate the experiment's execution by examining invariant metrics, also known as guardrail metrics. These metrics serve two primary purposes:

1. Trust-related Guardrails: Metrics such as sample size consistency and cache-hit rates ensure that the Control and Treatment groups align with the experiment configuration. Any deviation here might indicate issues in randomization or assignment.

2. Organizational Guardrails: Metrics like latency or system performance, which are crucial to business operations, should remain stable across variants. For example, significant changes in checkout latency would signal underlying problems unrelated to the coupon code test.

If these metrics show unexpected changes, it suggests flaws in the experiment design, infrastructure, or data processing pipeline. Addressing these issues before analyzing the core results is vital to maintaining trust in the findings.

Once the guardrails are validated, the next step is to analyze and interpret the results with precision. For example, if the p-value for revenue-per-user in both Treatment groups is below 0.05, we reject the null hypothesis and conclude that the observed differences are statistically significant. However, statistical significance alone is insufficient. Practical significance—the magnitude of the observed effect—determines whether the change is worth implementing.

Results Table

From the table, we observe that both treatments significantly reduce revenue-per-user compared to the control group. While the p-values confirm statistical significance, the negative impact on revenue highlights the need to reassess introducing coupon codes.

Decision-Making Framework

Consider the context of your experiment:

• Short-Term vs. Long-Term Impact: Changes with minimal downside risks, such as testing promotional headlines, may allow for lower thresholds of significance. Conversely, introducing high-cost features like a coupon code system requires higher thresholds due to long-term resource commitments.

• Balancing Metrics: A decrease in revenue may be acceptable if offset by an increase in user engagement, but only if the net impact aligns with organizational goals.

Ultimately, the results must translate into a clear decision framework. For example:

1. If the results are both statistically and practically significant, the decision to launch is straightforward.

2. If statistically significant but not practically meaningful, the change may not justify further investment.

3. If results are inconclusive, consider increasing sample size or re-evaluating the design.

4. The result is statistically significant, and likely practically significant. Like prior examples, it is possible that the change is not practically significant. In this situation, repeating the test with greater power is advisable. However, from a launch/no-launch perspective, choosing to launch is a reasonable decision. It is crucial to explicitly document the factors influencing this decision, particularly how they align with the practical and statistical significance boundaries. This clarity not only supports current decision-making but also establishes a solid foundation for future analyses.

By grounding decision-making in robust analysis and broader business considerations, organizations can confidently use A/B testing as a tool for sustainable growth and innovation.

References: ASA statement on p-values; Microsoft’s sample-ratio mismatch guidance.

"One accurate measurement is worth more than a thousand expert opinions."

– Admiral Grace Hopper

In the digital age, where data reigns supreme, the ability to measure the impact of changes accurately is crucial. The famous quote by Admiral Grace Hopper encapsulates the essence of why businesses like Microsoft's Bing have turned to A/B testing to drive innovation and profitability.

The Bing Case Study

In 2012, a small yet transformative idea within Microsoft’s Bing team provided a striking example of the power of data-driven experimentation. A simple suggestion to change ad headlines was overlooked for months, buried under bigger projects. When finally implemented, this idea became Bing’s most significant revenue-generating change, highlighting the profound impact of rigorous online controlled experiments (A/B tests).

The proposed change involved extending ad title lines by combining them with text from the line below, creating a more engaging headline. Initially deemed low priority, it wasn’t until a developer implemented it—due to its coding simplicity—that the idea was tested on real users. A/B testing split users between the existing and modified layouts, tracking interactions, clicks, and revenue metrics.

Within hours of launch, an anomaly emerged: revenue surged unexpectedly. Teams braced for a potential bug, but upon verification, the results proved legitimate. The new ad layout increased Bing’s revenue by 12%, equating to over $100 million annually in the U.S. alone, highlighting several key lessons:

• Value Assessment: Even seemingly minor ideas can have significant impacts, yet their value is often underestimated or overlooked.

• Impact of Small Changes: A small alteration can lead to massive financial returns if it aligns well with user behavior and expectations.

• Rarity of Big Wins: Not every change will yield such dramatic results; this was one of the few blockbuster successes from thousands of experiments conducted annually.

• Efficiency in Experimentation: The overhead of running an experiment must be small. Bing’s engineers had access to ExP, Microsoft’s experimentation system, which made it easy to scientifically evaluate the idea.

What is A/B Testing?

A/B testing, also known as split testing, is a method of comparing two versions of a webpage, email, or other marketing asset to determine which one performs better. By splitting your audience into two groups and showing each group a different version (Version A vs. Version B), you can measure which variation drives more conversions, clicks, or other desired outcomes.

Companies like Airbnb, Amazon, Booking.com, eBay, Facebook, Google, LinkedIn, Lyft, Microsoft, and Netflix rely heavily on online controlled experiments (Gupta et al., 2019). These organisations conduct thousands to tens of thousands of experiments annually, often involving millions of users. They test a wide range of elements, such as user interface (UI) changes, relevance algorithms (including search, ads, personalization, and recommendations), latency and performance, content management systems, customer support systems, and more. Experiments span various channels, including websites, desktop applications, mobile apps, and email.

For example, if you’re testing a landing page, you might experiment with two different headlines. Group A sees “Save Big on Your Next Adventure,” while Group B sees “Discover Affordable Travel Deals.” By analysing which headline leads to more sign-ups or sales, you gain valuable insights into what resonates with your audience.

In typical online controlled experiments, users are randomly assigned to different variants in a consistent manner, ensuring they experience the same variant across multiple visits. In the Bing example, the Control group saw the original ad display, while the Treatment group saw ads with longer titles. User interactions with the Bing website were tracked and logged, enabling the calculation of metrics from the logged data. These metrics were then used to evaluate the differences between the variants.

Understanding A/B Testing Terminology

A/B testing, or controlled experiments, involves several terms:

• Overall Evaluation Criterion (OEC): A metric that encapsulates the experiment's goal, like increase in revenue balanced with user experience metrics. The OEC should be measurable within the short timeframe of an experiment while being expected to causally influence long-term strategic goals.

  Experiments may have multiple objectives, and analysis can employ a balanced scorecard approach. However, it is highly recommended to select a single metric, potentially as a weighted combination of these objectives, to streamline evaluation (Roy 2001, 50, 405−429).

• Parameters: A controllable experimental variable, often called a parameter, is a factor that can be adjusted during an experiment and is believed to influence the Overall Evaluation Criterion (OEC) or other key metrics of interest. These parameters are assigned specific values or levels, representing the variations being tested. Understanding how these parameters affect outcomes allows experimenters to optimize for the best results.

  • Simple A/B Tests:
    In an A/B test, there is typically one parameter with two levels. A single parameter like button color might have two levels: blue (A) and red (B), with the goal of determining which generates higher click-through rates.

  • Univariable Tests with Multiple Levels:
    A test with a single parameter that has more than two levels.A parameter like button placement could have levels such as top of the page (A), middle (B), bottom (C), or sidebar (D), to identify the most effective location.

  • Multivariable Tests (Multivariate Tests - MVTs):
    MVTs test multiple parameters simultaneously, making it possible to analyze how their interactions impact results.Multiple parameters are tested together, such as button shape (round, square) and button text (“Buy Now,” “Add to Cart”), to evaluate combinations and uncover interactions that maximize conversions.

  While simple A/B tests or univariable designs are effective for evaluating straightforward changes, MVTs are particularly useful when multiple factors interact in non-obvious ways. For instance, a font size that performs well with one color might perform poorly with another. By testing these combinations, experimenters can discover a global optimum—the best overall combination of changes that maximizes the desired outcome.

• Variants: A user experience being tested is defined by assigning specific values to parameters, creating distinct variants for comparison. In a typical A/B test, these variants are labeled as Control and Treatment, with the Control representing the existing version (baseline) and the Treatment reflecting the modified version being tested. While some literature uses "variant" solely for the Treatment, the Control is also considered a critical variant, serving as the benchmark for measuring changes in key metrics.

  For example, in an experiment testing button colors, the Control might feature the current blue button, while the Treatment introduces a red button. If a bug or unexpected issue arises during the experiment, it is standard practice to abort the test and ensure all users are reverted to the Control variant, thereby minimizing the potential for adverse impacts. This process safeguards the user experience while maintaining the integrity of the baseline performance data for future analysis.

• Randomisation Unit: Randomization is a critical component of controlled experiments. A pseudo-randomization process, such as hashing, is applied to units (e.g., users or pages) to assign them to different variants. Proper randomization ensures that the populations assigned to each variant are statistically similar, enabling causal effects to be determined with high confidence. The mapping of units to variants must be both persistent and independent. For example, if a user is the randomization unit, that user should consistently experience the same variant throughout the experiment, and their assignment should reveal nothing about how other users are assigned.

  Using users as the randomization unit is highly recommended for online experiments targeting audiences across websites, apps, or other digital platforms. However, alternative randomization units are sometimes employed based on the experiment's goals. These can include:

  • Pages: Randomising content displayed on specific pages.

  • Sessions: Assigning a variant to a single user session but allowing different experiences across sessions.

  • User-Days: Ensuring a consistent experience for a user within a specific 24-hour period defined by the server.

  Proper randomisation is essential to maintain the integrity of the experiment. In cases where each variant is assigned an equal proportion of users, every user must have an equal probability of being assigned to any variant. This deliberate process eliminates potential biases that could distort results.

Why is A/B Testing Important?

The challenge with data interpretation lies in distinguishing correlation from causation. Observational data can mislead; for instance, users experiencing more errors might show lower churn rates, not because errors are beneficial, but because they are heavy users. Controlled experiments help establish causality by systematically varying one thing at a time while keeping everything else constant.

Example: Microsoft Office 365

Consider the case of Microsoft Office 365, a subscription-based software service. Observational data might reveal that users who encounter more error messages and software crashes have lower churn rates compared to others. At a glance, one might conclude that introducing more errors or lowering the software's quality could reduce customer churn, which is clearly not the logical step to take.

• Correlation: The data shows a correlation between seeing error messages, experiencing crashes, and lower churn rates.

• Misleading Interpretation: One might mistakenly infer that errors or crashes somehow keep users engaged or less likely to unsubscribe.

• Actual Causation: However, the real underlying factor here is "usage". Heavy users of the product are more likely to encounter errors due to the frequency and intensity of their use. These heavy users also tend to have lower churn rates because they find more value in the product or are more invested in it, not because the errors are beneficial.

This example with Microsoft Office 365 illustrates why jumping to conclusions from mere correlations can lead to flawed strategies. Controlled experiments provide a structured approach to validate hypotheses, ensuring that decisions are based on causal relationships rather than coincidental correlations. By isolating variables and observing the direct impact of changes, businesses can avoid costly mistakes and invest in strategies that genuinely contribute to better user experiences and business outcomes.

Key Elements of an A/B Test

For A/B testing to be effective:

1. Experimental Units (or Users): Must be assignable to different variants without interference.

2. Sufficient Scale: Enough units to detect even small effects statistically.

3. Metrics: Key metrics like an OEC must be clearly defined and measurable.

4. Ease of Change: Software changes should be implementable without significant overhead.

Subscribe now

Lessons for Organizations

1. Data-Driven Decision Making: Establish a clear OEC that aligns with strategic goals, ensuring measurable and actionable insights.

2. Invest in Experimentation Infrastructure: Reliable systems are crucial for managing experiments, logging data, and deriving trustworthy conclusions. Also, implementing software changes with minimal overhead is a critical enabler for effective experimentation. Agile systems and processes should allow teams to make swift updates, test hypotheses, and adapt based on findings. This approach reduces delays, fosters innovation, and ensures experiments can iterate rapidly to achieve meaningful insights without bottlenecks.

3. Embrace Humility in Idea Assessment: Most ideas fail to deliver the anticipated impact. Controlled experiments help validate assumptions, reducing wasted effort and guiding resources toward high-value initiatives.Common A/B Testing Mistakes to Avoid

Strategic and Tactical Benefits

• Strategy Validation: Experiments can affirm or challenge strategic directions through tangible outcomes.

• Tactical Optimization: Small, iterative changes can lead to substantial cumulative gains.

• Pivoting: When experiments hint at strategic misalignments, they can lead to strategic pivots or reassessments.

Conclusion

The Bing example serves as a powerful illustration of how A/B testing can revolutionize digital products and experiences. It highlights the importance of fostering a culture that prioritizes data-driven decision-making, backed by robust infrastructure to conduct rigorous experiments at scale. Moreover, it emphasizes the value of intellectual humility—recognizing that even the most promising ideas must be validated through empirical evidence, as our initial intuitions are often flawed.

As digital platforms continue to evolve in complexity and scale, the reliance on controlled experiments will become even more indispensable. These experiments not only validate innovations but ensure they deliver measurable value to both businesses and users. By systematically testing and learning, organizations can refine their offerings, reduce risks, and make decisions grounded in objective insights rather than assumptions. In this way, A/B testing acts as a critical enabler of innovation, ensuring that progress is not only creative but also meaningful and impactful.

Reference

1. Kohavi, Ron, Diane Tang, and Ya Xu. 2020. ​Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.

2. Gupta, Somit, Ronny Kohavi, Diane Tang, Ya Xu, and etal. 2019. “Top Challenges from the first Practical Online Controlled Experiments Summit.” Edited by Xin Luna Dong, Ankur Teredesai and Reza Zafarani. SIGKDD Explorations (ACM) 21 (1). https://bit.ly/OCESummit1.

3. Roy, Ranjit K. 2001. Design of Experiments using the Taguchi Approach : 16 Steps to Product and Process Improvement. John Wiley & Sons, Inc.

Meta prioritise long-term satisfaction over immediate interactions. Their goal is to ensure users connect and find value in their Facebook experience. But figuring out which posts deliver lasting value is tricky.

Consider a post about a local volunteering opportunity – a user might get inspired, join, and feel fulfilled over time. On the other hand, a funny meme might get a quick laugh when scrolled past. Which contributes more to the user's long-term satisfaction? Meta uses surveys to ask users directly, but these have limitations in terms of survey volume and imperfections within them. So, they also rely on data analysis and statistics to estimate which content truly enriches users' experiences on Facebook over time.

The Experiments

To leverage the power of data, Meta employs A/B testing, a widely used method in the industry to understand cause and effect. It begins with crafting meticulous experiments involving multiple variants. For instance, one experiment might expose some Facebook users to more health and wellness-related posts, while others might see increased content about technology and innovation.

It's crucial to encompass all content types existing on the platform within these experiments. This means testing the shift in distribution for every major content category. The key is to maintain broad content definitions, such as amplifying 'content about hobbies' rather than specific subcategories like 'travel photos.' This approach allows for a diverse range of content without requiring an excessive number of experiments.

Through a sequence of experiments, the team evaluates user value in each test over an extended period, often spanning months. This extended duration allows them to understand how various content types relate to long-term user value. Upon discerning these relationships, determining which content deserves higher ranking becomes more straightforward.

The Meta-Analysis Method

After the completion of these experiments, Meta undertakes a comprehensive analysis utilising an experiment meta-analysis method. This method refers to a statistical technique used to analyse and synthesise findings from multiple independent studies or experiments. It involves consolidating the outcomes and results obtained from various experiments conducted to assess user behaviour or satisfaction.

In this method, the emphasis lies on long-term metrics such as Customer Lifetime Value and retention rates. For instance, let’s consider the M-1 retention metric—a metric that quantifies user continuity across consecutive months in platform engagement. This metric offers insights into user persistence, revealing the platform's ability to sustain ongoing user interest and consistent interaction over time.

Let’s denote change in M-1 retention metric as Y₁%, Y₂%, and so on, across multiple experiments. For instance, if change in M-1 retention denoted by Y₁%, it signifies the change in user retention percentage across the first experiment, emphasising relative changes rather than absolute numerical values.This analysis aids in understanding how alterations implemented in the experiments influence user behaviour and satisfaction over time, as reflected in metrics like M-1 retention.

In each experiment, we examine how the mix of content shifts as a result of the design of the experiment. For instance, if there's a 15% increase in news articles displayed, it could mean a slight decrease in other content categories. Remember, it's not just one type of content that changes but multiple types together.

For illustration, let's say in the first experiment, there's a change of x₁₁% in news articles (where '1' represents news content, and the second '1' signifies the first experiment), a shift of x₂₁% in videos, and x₃₁% in user-generated posts. Similarly, subsequent experiments might show variations like x₁₂%, x₂₂%, x₃₂%, and so forth. Here, x can be negative as well.

Subsequently, a straightforward linear regression is performed on the aggregated data at the percentage per treatment level. The regression equation is:

where the coefficients (coeff ₖ) are determined as the elasticities representing the impact of increasing or decreasing the distribution of specific content type xₖ in the experiments.

Interpreting the coefficients

If a coefficient (coeff ₖ) is statistically significant and positive, it indicates that showcasing more of that specific content type to users (while keeping all other content types constant) would likely lead to a positive impact on user value.

For example, suppose we're analyzing the impact of increasing video content (let's call it x₁) in our experiments. The coefficient (coeff₁) associated with x₁ indicates how much the increase in videos affects the observed outcomes. If coeff₁ is positive and significant, it implies that showcasing more videos might positively impact M-1 retention, encouraging users to stay engaged or active over consecutive months.

Additional Pointers

For a more resilient analysis, it's essential to consider and apply these outlined pointers.

• The ideal scenario necessitates as many experiments (n) as independent variables (k), ensuring accurate estimations; thus, k should be considerably lesser than n.

• To generate new experiments for analysis, diverse treatment strengths, such as altering content distribution by varying percentages (e.g. in one experiment we increase the distribution of meme content by 10%, in another by 20%)), or conducting experiments at different times, prove beneficial.

• They noted that conducting a smaller number of clear experiments, involving changes in all content types in at least one trial, works effectively.

• Starting with a simple linear regression serves as a solid starting point before delving into more complex ML models.

• The linear regression model mentioned earlier assumes constant elasticities. This means that a 1% increase in viewing a specific content type results in a y% change in the outcome variable. It's assumed that a 2% increase in viewing leads to a 2 * y% change. This constant elasticity has been observed at Facebook, scaling consistently across users with varying activity levels.

In summary, Meta's methodology for optimising the Facebook Feed is a systematic process based on comprehensive experimentation, data analysis, and statistical modelling. By prioritising long-term user satisfaction over immediate engagement metrics, they conduct numerous experiments, utilising A/B testing and regression analyses to determine the impact of different content types on user value, often measured by metrics like M-1 retention. Their approach involves examining content shifts, interpreting coefficients from linear regression models, and ensuring robust experiment designs with various treatment strengths. Ultimately, this data-driven approach helps Meta tailor and refine the Facebook Feed to better match users' preferences, creating a more engaging and satisfying experience for everyone.

Why Quality-Based User Segmentation Is Essential?

Segmenting users based on their quality offers numerous strategic advantages. One key benefit lies in crafting precise user targeting. By discerning the quality of users, you can categorise them into distinct segments, such as high-quality, mid-range, or low-quality users. Each of these segments can then be approached with tailored strategies.

• For high-quality users, you can design loyalty programs that provide exclusive benefits, nurturing a sense of belonging and engagement.

• Mid-range users might receive targeted offers that match their preferences, incentivising them to remain active and engaged.

• Low-quality users, on the other hand, may benefit from re-engagement efforts to improve their experience and potentially transform them into higher-quality users.

In essence, quality-based segmentation forms the foundation for strategies that optimise user engagement, foster loyalty, and drive business growth.

The User Quality Score: How We Calculate It

Defining the quality of users begins with identifying and measuring their engagement with your platform. This typically involves pinpointing specific, high-value actions (HVAs) that are vital to your business goals. HVAs are actions like "add to cart" or any other significant interaction that drives value. By monitoring both frequency and the variety of HVAs a user engages in, we can assess their quality.

Calculating the quality score often involves taking a weighted average of the number of times a user performs these HVAs. This weighted average considers the significance of each action; for example, "checkout" might be given more weight than a "product view" because it represents a closer step to conversion.

Assigning Weights to High-Value Actions

Defining the weights for high-value actions (HVAs) involves a systematic process. We begin by determining a "conversion event," which is the specific action that signifies a user's successful journey or achievement of a significant goal on our platform.

To assign weights to HVAs, we analyze a set of new users and track how many of them perform a particular HVA before reaching the conversion event. The weight for an HVA is calculated by taking the ratio of users who performed that action before conversion to the total number of users who successfully converted.

For example, let's say 100 new users successfully converted. Among them, 60 users engaged in a "search" action before converting. This results in a weight of 0.6 for the "search" action. In contrast, only 30 users visited the "product description" section before conversion, resulting in a weight of 0.3 for this action. This process allows us to assign weights based on the influence of each HVA on user conversions.

Calculating the Final User Quality Score

To determine the final user quality score, we employ a straightforward yet effective method. We start by calculating the sum product of the weight assigned to each high-value action (HVA) and the number of times those HVAs are performed by a user. This gives us a raw score that reflects a user's overall engagement and the significance of their actions.

To account for the frequency of these actions, we implement a normalization process. Instead of relying on absolute numbers, we use percentile distribution among new users who engage in these HVAs before converting. For instance, if 20% of users perform less than 4 searches before converting, then the normalized value for 4 searches would be 0.2. This normalisation ensures that user quality scores are relative and not skewed by extreme values, allowing for a more balanced and accurate assessment.

Utilising Quality Score Buckets for Personalisation and Targeting

These quality scores can then be segmented into various buckets or categories. Each bucket represents a range of quality scores, allowing us to group users with similar levels of engagement and actions. These score buckets become invaluable for personalisation and targeting strategies.

Users falling into the same score bucket can receive tailored content, offers, and messages that suit their specific engagement and preferences. This personalisation enhances the user experience, boosts engagement, and increases the likelihood of achieving desired outcomes. It also streamlines marketing efforts by ensuring that promotions and messages are sent to users who are most likely to respond positively. Overall, these quality score buckets serve as a powerful tool for optimising user interactions and achieving business goals.

Hey there! Piyush here, and I'm thrilled to bring you another edition of my weekly analytics newsletter. Let's dive right in!

In the first part of our case study, we explored the initial stages of our investigation into the gradual drop in conversion rate for a fashion e-commerce giant. We witnessed how Rohit, our diligent product analyst, leveraged cohort and funnel analysis to gain valuable insights into the user journey. Now, we dive deeper into the next phase of our investigation, where we uncover conclusive findings and take action to address the identified issues.

To revisit the first part of this blog, please refer to the link provided.

As we left off, Rohit had discovered that the drop-off in conversion rate was most prominent for users who reached the product page through search on the app. This insight raised suspicions about potential issues with the app's search functionality or the relevance of search results.

The Analytics-Data Science Collaboration

In the vibrant and art-adorned office of the fashion e-commerce giant, Rohit crossed paths with Shreyash, the seasoned data scientist leading the search team.

They had crossed paths on several occasions but had never engaged in a detailed conversation. Little did they know that their encounter would mark the beginning of a remarkable collaboration.

As Rohit approached Shreyash's desk, he noticed the walls adorned with intricate graphs and equations, a testament to Shreyash's expertise in data science. With a slight hint of nervousness, Rohit cleared his throat and initiated the conversation.

Rohit: "Hi Shreyash, I hope I'm not interrupting anything important. I've been diving deep into our app's user journey, specifically focusing on the drop in conversion rate for users who reach the product page through search."

Shreyash, known for his calm demeanour, looked up from his laptop screen and offered a friendly smile.

Shreyash: "Not at all, Rohit. I'm always open to discussions that can help us uncover insights and improve our systems. Please, have a seat. What have you discovered so far?"

Rohit took a seat, a sense of anticipation building within him. He began sharing his findings, outlining the peculiar drop-off in conversion rate and the potential issues with the search functionality or relevance of search results.

Rohit: "As I analysed the data, it became apparent that users who relied on the search feature were experiencing a significant drop in their conversion rate. This led me to suspect that there might be some issues with our search algorithm or the relevance of the search results."

Shreyash listened attentively, intrigued by Rohit's observations. He leaned back in his chair, ready to contribute his expertise.

Shreyash: "I appreciate your thorough analysis, Rohit. However, I must inform you that we recently deployed a new search model aimed at enhancing the user experience. The initial feedback and performance indicators have been promising. In fact, let me show you a few key search metrics that demonstrate the positive impact of our latest implementation."

Shreyash navigated through his meticulously organised dashboards and presented a set of insightful visuals, showcasing the improvement in search metrics.

Analysing Key Search Metrics

Shreyash proceeded to share the key search metrics that shed light on the performance of the search results. With a confident smile, he presented a set of insightful visuals, showcasing the following metrics:

1. Click-through Rate (CTR): This metric measures the percentage of users who click on the search results out of the total number of users who view them. Shreyash displayed a graph depicting a significant increase in the CTR for users on the new search model compared to users on the old model. This indicated that the new search algorithm was effectively driving higher engagement and click-through rates.

2. Conversion Rate: The conversion rate metric measures the percentage of users who complete a desired action, such as making a purchase or signing up for a service, out of the total number of users who interacted with the search results. Another graph demonstrated a noticeable improvement in the conversion rate of users on the new model compared to users on the old model. This suggested that the new search algorithm was more successful in driving desired actions and achieving business objectives.

3. Bounce Rate: Bounce rate measures the percentage of users who leave the app after viewing a single search result. A high bounce rate may indicate that the search results are not meeting user expectations or are not relevant to their needs. The graph displaying the bounce rate showcased a significant decrease for users on the new model compared to users on the old model. This indicated that the new search algorithm was successful in reducing user bounce rates, implying improved search result relevance and user satisfaction.

4. Zero-Result Rate: This metric measures the percentage of searches that yield no results. A lower zero-result rate indicates that the search algorithm is effectively returning relevant results for a wide range of user queries.

In addition to these metrics, Shreyash introduced the Mean Reciprocal Rank (MRR) as a metric to evaluate the average ranking of the search result clicked by users. MRR takes into account the order of the clicked search results and provides a measure of the effectiveness of the search algorithm in ranking the most relevant results at the top. A higher MRR score indicates a better search result ranking and greater search result relevance.

Shreyash explained further, "MRR is calculated by taking the reciprocal of the rank of the first relevant search result clicked by the user. For example, if a user clicks on the most relevant result as the third item displayed, the reciprocal rank would be 1/3. A higher MRR score indicates that users are finding more relevant results higher up in the search rankings."

Rohit's suspicions about the search functionality and relevance began to dissipate as the metrics provided evidence of the new search model's effectiveness. Shreyash's analysis compared users on the new search model to users on the old model, revealing positive improvements in CTR, conversion rate, average time on page, and reduced bounce rates.

As the conversation progressed, Rohit couldn't help but express his concern about the potential biases in the data. He recognised that the new search model had not undergone an A/B test, which meant there could be hidden influences impacting the results. In an effort to ensure a comprehensive and unbiased analysis, Rohit shared his thoughts with Shreyash.

"Shreyash, while the metrics indicate positive results, we should be cautious about potential biases in the data. Without an A/B test, we need to carefully examine user cohorts and segment-specific behaviours to uncover any hidden influences on the drop in conversion rate. It's crucial that we dive deeper and gain a comprehensive understanding to make informed decisions," Rohit explained.

Shreyash listened attentively, understanding the importance of addressing potential biases and conducting a thorough analysis. They both agreed that exploring user cohorts, segment-specific search experiences, and delving deeper into the data would be essential to gain a clearer picture of the search dynamics and optimise the search experience for all users.

With a shared commitment to uncovering the truth and refining the search, Rohit and Shreyash embarked on the next phase of their investigation, eager to unveil the underlying factors that contributed to the gradual drop in conversion rate.

Unmasking the Bias: Investigating User Behaviour and App Adoption

As Rohit delved deeper into the analysis, he aimed to examine the possibility of any bias influencing the results. To assess this, he shifted his focus to user engagement and retention metrics that were not directly influenced by the search model.

Rohit scrutinised metrics such as sessions per user, the mix of new versus repeat users, and the average age of users on the platform. These metrics would help him understand if there were any inherent differences between the two cohorts: users on the new search model and users on the old model.

To his surprise, Rohit noticed a significant disparity between the two cohorts. Users on the new search model exhibited higher levels of engagement and transactional activity on the platform. However, Rohit questioned whether this improvement was solely due to the new model's performance or if it was a result of selection bias.

He realised that the users on the new model might appear to have better user metrics not because of the model's performance but because of the selection bias. It was possible that the users on the new model were of better quality to begin with, leading to better metrics.

Continuing his investigation, Rohit delved further into understanding the source of the observed bias. He carefully examined the circumstances surrounding the deployment of the new search model and its impact on user behaviour.

Rohit soon realised that the new model was introduced with a major app release, which included updates and enhancements beyond just the search functionality. As a result, users would only receive the new model if they actively updated their app to the latest version. This realisation led him to a significant insight—the bias stemmed from the fact that only active users, who were more likely to engage and transact on the platform, had updated to the new app version and were consequently using the new search model.

The bias, therefore, originated from the self-selection of users who actively updated their app. These users represented a segment of highly engaged and committed individuals, which naturally influenced the user metrics and potentially skewed the performance evaluation of the new search model.

With this newfound understanding, Rohit successfully identified the underlying cause of the bias. He gained a valuable insight into the significance of considering user behaviour and adoption patterns when evaluating the performance of the new model. This realisation prompted Rohit and Shreyash to recognise the importance of implementing an AB experiment, which would provide a robust framework for unbiased analysis and enable them to draw causal inferences.

They understood that conducting an AB experiment was crucial for any new updates or enhancements to negate the impact of biases and accurately assess the effectiveness of changes made. By incorporating this experimental approach, Rohit and Shreyash aimed to establish a causal relationship between the new search model and its impact on user engagement and conversion rates.

The AB Test: Deriving Causal Inference

Excited by their newfound insights and armed with the understanding of the bias, Avantika, Rohit, and Shreyash embarked on a crucial next step: conducting an AB experiment. They understood that this experiment would provide a rigorous and unbiased evaluation of the new search model's performance. With careful planning and implementation, they executed the experiment, comparing the user engagement and conversion rates between the users on the new search model and a control group on the old model.

To their surprise, the results of the AB experiment revealed that the new search model performed poorly in terms of user engagement and conversion rates. This outcome validated their initial concerns and highlighted the importance of thorough evaluation and experimentation before implementing major updates. Armed with this knowledge, Avantika, Rohit, and Shreyash were able to make informed decisions for optimising the search experience and driving improved conversion rates.

Key Discoveries and Conclusive Actions

The collaborative efforts of Avantika, Rohit, and Shreyash highlighted the importance of leveraging data-driven insights, conducting thorough analysis through funnel and cohort examination, and embracing experimentation to drive innovation and enhance user experiences. By meticulously dissecting the user journey through funnel analysis, they gained valuable insights into the specific stages where the drop in conversion rate occurred, enabling them to identify the root cause with precision. Furthermore, by segmenting users into cohorts and analysing their behaviours, they were able to uncover hidden influences and nuances that contributed to the overall performance of the search model.

As they concluded their investigation, Avantika, Rohit, and Shreyash eagerly shared their findings and recommendations with the wider team, stressing the significance of funnel and cohort analysis in understanding user behaviour and identifying areas for improvement. They emphasised the need for ongoing monitoring and analysis of key metrics throughout the user journey to ensure a comprehensive understanding of the entire conversion funnel.

In addition to their focus on funnel and cohort analysis, Avantika, Rohit, and Shreyash recognised the vital role of AB experimentation in deriving causal inferences and drawing accurate conclusions. They highlighted the importance of conducting controlled experiments, where users are randomly assigned to different versions of the search model, to measure the true impact of any changes made. By implementing AB testing, they could negate biases and confidently evaluate the performance of the new search model, ultimately leading to more informed decision-making and continuous improvement.

Avantika, Rohit, and Shreyash's collaborative efforts established a framework for unbiased evaluation, hypothesis testing, and deriving causal inferences through AB experimentation. Their achievements have set a precedent for future endeavours, ensuring that funnel and cohort analysis, coupled with rigorous AB testing, will remain integral to the company's growth and success in the dynamic world of online retail.

Share

Context note — 3 October 2026. This is a fictional teaching example. The characters, business scenario and numerical examples are illustrative; they are not a report using data from any employer. It is retained in Measurement Guides and predates Inferentia’s research standards.

It was a rainy evening in Bangalore, and Avantika, the Head of Product at an Indian e-commerce giant specialising in fashion and lifestyle products, was meeting with her analytics counterpart, Rohit. As they sipped on their hot cups of tea, Avantika voiced her concern about the gradual drop in session level conversion rate compared to last month. Rohit quickly pulled up the conversion dashboard and confirmed Avantika's observation. They both knew that a drop in conversion rate could significantly impact the company's bottom line, and it was crucial to investigate the root cause of the problem.

"Have you noticed any changes in the mix of our cohorts?" Rohit asked, looking over the data. "Perhaps the new users aren't converting as well as the old ones."

"That's a good point," Avantika replied, "But we haven't made any significant changes to our acquisition strategy recently."

"What about any recent changes to the product or the user experience?" Rohit asked, scrolling through the data.

"I don't think so," Avantika said, thinking hard. "But let's double-check."

Rohit nodded and promised to conduct a thorough analysis of the data to identify any potential issues.

Based on the initial conversation between Avantika and Rohit, they both had a few hypotheses about the possible cause of the drop in conversion rate. However, before jumping to any conclusions, they needed to conduct a thorough investigation of the data. Rohit's promise to conduct an analysis was just the beginning of a lengthy process that would require them to dive deep into the data and explore different variables.

Avantika and Rohit knew that this was not an easy task, but they were determined to get to the bottom of the issue. They realised that every aspect of their business, from the product and user experience to marketing and advertising, could potentially impact the conversion rate. Therefore, they needed to be meticulous in their approach and consider every possible angle.

Initial Analysis

Rohit knew that the issue could stem from various factors. He started by analysing the user experience of the app, but after going through the design and user interface, he found only minor issues that were unlikely to cause such a significant drop in conversion.

He then explored the pricing hypothesis but did not find any significant changes in pricing or discounts that could have led to the decline in conversion. He also investigated whether there was a change in the user mix, but found no significant differences in the demographic distribution of users during the period.

Additionally, Rohit ran various tests on the app's functionality to check if there were any technical issues that could have caused the drop in conversion rate, but found no glitches that could be responsible.

Even after thorough analysis, Rohit could not find any significant changes that could explain the decline in conversion rate. He was feeling increasingly frustrated and overwhelmed by the sheer volume of data that he had collected. He realised that without a clear structure in his analysis, he was getting lost in the sea of data and unable to make any meaningful conclusions.

Deconstructing the conversion rate: Cohort and Funnel Analysis

Rohit's experience highlights the importance of having a structured approach to data analysis. Without a clear structure, it's easy to get lost in the vast amount of data and lose sight of the goal. A structured approach helps to break down the problem into manageable pieces and allows for a more systematic investigation of potential causes.

Breaking down metrics into different dimensions is essential for a systematic and thorough investigation of potential causes for any issue or problem. This deconstruction allows for a more in-depth analysis of different aspects of the business, making it easier to identify specific areas that might be contributing to the problem.

Cohort and funnel analysis are two dimensions that are particularly useful in investigating the root causes of changes in conversion rate. Cohort analysis can be thought of as the vertical dimension as it enables businesses to understand how different user groups behave over time, allowing for the identification of specific user groups that might be contributing to the problem. Funnel analysis, on the other hand, can be seen as the horizontal dimension as it helps to identify which parts of the user journey might be causing issues, such as drop-offs in the user journey.

By breaking down the conversion rate into two dimensions, businesses can approach the problem in a more structured and targeted way. This can help to save time and resources that might otherwise be spent on investigating irrelevant factors or variables.

Moreover, by focusing on specific dimensions, businesses can develop more targeted and effective solutions to the problem. For instance, if the cohort analysis highlights that a specific user group is experiencing a lower conversion rate, businesses can develop specific strategies to address the issues faced by this group. Similarly, if the funnel analysis identifies that users are dropping off at a specific stage of the user journey, businesses can focus on improving that stage to reduce the drop-off rate.

Overall, breaking down metrics into different dimensions, such as cohort analysis and funnel analysis, is crucial for a systematic and thorough investigation of potential causes. This approach enables businesses to identify specific areas that might be contributing to the problem, develop targeted solutions, and save time and resources in the process.

Subscribe now

Cohort Analysis

Rohit took a structured approach to analyse the conversion rate by deconstructing it into different user cohorts. He began by analysing cohorts based on user acquisition channels, including organic search, social media, and paid advertising. Rohit observed that the conversion rate had dropped across all channels, ruling out the possibility of acquisition issues.

Next, he looked into user type cohorts, such as new users and returning users. Rohit found that the drop in conversion rate was across new as well as returning users. But the quantum of drop was more pronounced among new users than returning users, which led him to investigate the onboarding process for new users and identify areas for improvement.

Further, Rohit analysed cohorts based on the device used by the users, such as desktop and mobile. He discovered that the conversion rate had dropped more on mobile devices than on desktops. This led him to investigate the mobile user experience and identify potential issues.

Moreover, Rohit analysed cohorts based on the user location, such as users from different regions or countries. He found that the conversion rate had dropped across all the regions.

Finally, Rohit analysed cohorts based on the product category, such as clothing, footwear, and accessories. He observed that the conversion rate had dropped more for certain product categories such as ethnic wear and personal care, leading him to investigate potential issues with those specific product categories.

Rohit's structured approach to analysing the conversion rate has provided him with a clear direction for further investigation. He has identified specific user groups that are contributing to the drop in conversion rate, such as new users, mobile users, and users of certain product categories. However, he still needs to pinpoint the exact source of trouble within these cohorts. This is where funnel analysis comes into play. By analysing the steps users take before completing a desired action, such as making a purchase, Rohit can identify where users are dropping off in the conversion funnel. This will help him identify specific issues that need to be addressed to improve the overall conversion rate.

Share

Funnel Analysis

A funnel analysis is a method of analysing a user's journey through a website or app by breaking it down into a series of steps, or stages, which are then analysed to identify any points of drop off or conversion.

Rohit looked at the trend line of the drop between consecutive steps in the funnel. This helped him to identify the exact point in the user journey where the drop-off was occurring and develop strategies to address the issue.

To perform a funnel analysis, Rohit began by defining the different steps in the user journey on the app. He identified the following stages:

• All Users: This is the total number of visitors to the app

• Non-bounced Users: These are users who did not leave the app after opening it

• Users with Product page view: These are users who clicked on a specific product to view its details

• Users with Add to Cart: These are users who added a product to their cart

• Users with Checkout: These are users who proceeded to the checkout process after adding a product to their cart

• Users with Orders: These are users who completed a purchase on the app

By analysing the funnel, Rohit could see how many users were dropping off at each stage of the journey. He observed that there was a significant drop between non-bounced users and users with a product page view. This drop-off is a critical metric to analyse because it represents the point where users start to engage with the app's content.

To gain a deeper understanding of the drop between non-bounced users and users with a product page view, Rohit decided to break down the user journey between the two stages into three different paths: users who reach the product page through search, users who reach the product page through product recommendation, and users who reach the product page through category navigation.

• Users who reach the product page through search: These are users who use a search engine or an internal search bar on the app to find a specific product. They enter relevant keywords or phrases into the search bar and click on the product that matches their search criteria.

• Users who reach the product page through product recommendation: These are users who are directed to a specific product through a recommendation algorithm on the app. The algorithm may use factors such as the user's past purchase history, browsing history, or items in their cart to suggest products that may be of interest to them.

• Users who reach the product page through category navigation: These are users who navigate through different categories and subcategories on the app to find the product they are interested in. They may use menus or filters to refine their search and eventually land on the product page.

After breaking down the user journey into these three paths, Rohit discovered that the drop-off was most prominent for users who reached the product page through search. This indicated that there may be issues with the app's search functionality or the relevance of search results.

Rohit decided to strengthen his hypothesis by overlaying the funnel analysis with user cohorts. His analysis revealed that the drop-off rate was notably higher for a specific group of users - new mobile users who were engaging with identified product categories. This information allowed Rohit to pinpoint the exact issue and concentrate his efforts on a particular cohort and segment of the user journey, leading to more targeted and effective solutions.

Rohit entered Avantika's office with a confident stride, holding a printout of the funnel analysis in his hand. He explained how he had broken down the user journey and identified the drop-off points, ultimately pinpointing the problem with the search algorithm for new mobile users engaging with specific product categories. Avantika was impressed with Rohit's findings and commended him on his thorough analysis. They both agreed to present the findings to the data science team responsible for the search algorithm and work on further deep dive.

In this first part of my blog, we explored how cohort analysis and funnel analysis can provide valuable insights into user behaviour on an app. Cohort analysis allows us to group users based on shared characteristics and observe how they behave over time, while funnel analysis helps us understand how users move through different stages of a conversion process.

We then saw how overlaying these two types of analyses can provide even deeper insights into user behaviour. By breaking down the user journey into different paths and overlaying the analysis by user cohorts, Rohit was able to pinpoint the problem areas in the app and focus his attention on improving the user experience for specific cohort and part of the user journey.

In the upcoming part of the blog, we will witness Rohit's efforts in concluding the analysis with definitive findings. We will observe how he handles a potential conflict between data science and analytics by utilizing the concept of causal inference. Along the way, we will learn about the significance of avoiding common pitfalls in analysis, such as selection bias, and the importance of conducting experiments to ensure accurate and actionable insights. Stay tuned for more exciting insights!

Subscribe now

Hello, I'm Piyush, and welcome to Inferentia - a platform where I share my insights and learnings on analytics in the e-commerce and fintech space. My focus is on various analytics verticals such as product, marketing, and growth analytics, and how they can transform businesses and drive growth.

In this first series of blog posts, we'll take a deep dive into The Cornerstones of Product Analytics, exploring the fundamental principles and strategies that underpin this essential discipline. Through this series, I will be taking you on a journey through the fascinating world of product analytics and showcasing how it has the power to transform businesses and drive growth. As someone who's worked extensively on product analytics, I've seen firsthand how it can be a game changer for organisations. From optimising product features to improving user experience, analytics plays a critical role in shaping business strategies. So join me on this exciting journey as we explore the world of product analytics together, and discover how data-driven insights can help you make smarter decisions and achieve better outcomes.

So what exactly is product analytics? At its core, it's the practice of collecting, analysing, and interpreting data to gain insights into how users interact with a product or service. For instance, product analytics can be used to understand how users interact with a website or mobile app, and identify areas for improvement. Let's say a company has an e-commerce website, and they notice that a large number of users abandon their shopping cart at the payment stage. Through product analytics, they can analyse user behaviour and identify the specific step that is causing the most drop-offs. This could be due to a complicated checkout process, lack of payment options, or other issues. With this information, the company can make targeted changes to improve the user experience and increase conversion rates.

Product analytics encompasses a wide range of techniques and methodologies that can be used to understand user behaviour, measure product performance, and identify areas for improvement. This involves a range of different components, including user engagement metrics, funnel analysis, cohort analysis, A/B testing, and predictive modeling, to name just a few. Each of these components plays a critical role in helping you gain insights into user behaviour, measure the impact of changes you make to your product, and ultimately drive growth for your business. Let's explore some of these areas in more detail:

Firstly, user engagement metrics are used to measure how users interact with a product or service. These metrics include things like time on site, click-through rates, and bounce rates. For example, if an e-commerce company notices a high bounce rate on their homepage, they may investigate and optimise the page to improve user engagement.

Funnel analysis is another important area of product analytics, where companies analyse the steps users take to complete a particular action or goal. This helps identify areas of the funnel that need improvement or optimisation. For instance, an online retailer may analyse their checkout funnel to understand why users are dropping off at certain steps and take steps to simplify the process.

Cohort analysis involves grouping users based on a common attribute, such as the month they signed up or the region they're from. By comparing the behaviour of different cohorts over time, companies can identify trends and make data-driven decisions. For example, a mobile app company may compare the retention rates of users who signed up in January versus those who signed up in February to understand how their product is performing. A/B testing is a method of comparing two versions of a product or feature to determine which performs better. For example, an email marketing campaign might test two subject lines to see which one results in a higher open rate.

Finally, predictive modeling uses historical data to make predictions about future outcomes. This can be used to inform decisions around everything from pricing to inventory management. For example, an online retailer may use predictive modeling to forecast demand for a particular product and adjust inventory levels accordingly.

In the upcoming posts, we will dive deep into each of these areas of product analytics, examining their core concepts, techniques, and best practices. We'll explore user engagement metrics, such as session duration, bounce rate, and retention rate, and discuss how they can help us understand user behavior and optimize product features. We'll also cover funnel analysis, which involves tracking user behavior as they move through various stages of the product journey and identifying key drop-off points. Cohort analysis, which allows us to group users based on common characteristics and track their behaviour over time, will also be explored in detail.

Furthermore, we will delve into A/B testing, a critical tool for testing product features, messaging, and pricing strategies. Predictive modeling, which involves using statistical algorithms to forecast future user behaviour, will also be a key area of focus.

We've only just scratched the surface of what product analytics has to offer. Over the course of this series, we'll take a deep dive into each of these areas, exploring best practices, case studies, and real-world examples to help you apply these concepts. And, of course, we'll cover much more than just the areas we've discussed today – from user engagement metrics to predictive modeling and beyond. So, if you want to stay up-to-date with the latest insights and strategies for leveraging data to drive growth, be sure to subscribe to our blog and join us on this exciting journey through the world of product analytics!