How Meta leverages the analytics to customise your Facebook feed?
A deep dive into Meta's advanced analytics and experiment framework
In this blog, we'll uncover how Meta uses analytics to personalise your Facebook Feed. We will talk about how Meta fine-tunes the Facebook Feed to better resonate with your interests and preferences.
Meta prioritise long-term satisfaction over immediate interactions. Their goal is to ensure users connect and find value in their Facebook experience. But figuring out which posts deliver lasting value is tricky.
Consider a post about a local volunteering opportunity – a user might get inspired, join, and feel fulfilled over time. On the other hand, a funny meme might get a quick laugh when scrolled past. Which contributes more to the user's long-term satisfaction? Meta uses surveys to ask users directly, but these have limitations in terms of survey volume and imperfections within them. So, they also rely on data analysis and statistics to estimate which content truly enriches users' experiences on Facebook over time.
The Experiments
To leverage the power of data, Meta employs A/B testing, a widely used method in the industry to understand cause and effect. It begins with crafting meticulous experiments involving multiple variants. For instance, one experiment might expose some Facebook users to more health and wellness-related posts, while others might see increased content about technology and innovation.
It's crucial to encompass all content types existing on the platform within these experiments. This means testing the shift in distribution for every major content category. The key is to maintain broad content definitions, such as amplifying 'content about hobbies' rather than specific subcategories like 'travel photos.' This approach allows for a diverse range of content without requiring an excessive number of experiments.
Through a sequence of experiments, the team evaluates user value in each test over an extended period, often spanning months. This extended duration allows them to understand how various content types relate to long-term user value. Upon discerning these relationships, determining which content deserves higher ranking becomes more straightforward.
The Meta-Analysis Method
After the completion of these experiments, Meta undertakes a comprehensive analysis utilising an experiment meta-analysis method. This method refers to a statistical technique used to analyse and synthesise findings from multiple independent studies or experiments. It involves consolidating the outcomes and results obtained from various experiments conducted to assess user behaviour or satisfaction.
In this method, the emphasis lies on long-term metrics such as Customer Lifetime Value and retention rates. For instance, let’s consider the M-1 retention metric—a metric that quantifies user continuity across consecutive months in platform engagement. This metric offers insights into user persistence, revealing the platform's ability to sustain ongoing user interest and consistent interaction over time.
Let’s denote change in M-1 retention metric as Y₁%, Y₂%, and so on, across multiple experiments. For instance, if change in M-1 retention denoted by Y₁%, it signifies the change in user retention percentage across the first experiment, emphasising relative changes rather than absolute numerical values.This analysis aids in understanding how alterations implemented in the experiments influence user behaviour and satisfaction over time, as reflected in metrics like M-1 retention.
In each experiment, we examine how the mix of content shifts as a result of the design of the experiment. For instance, if there's a 15% increase in news articles displayed, it could mean a slight decrease in other content categories. Remember, it's not just one type of content that changes but multiple types together.
For illustration, let's say in the first experiment, there's a change of x₁₁% in news articles (where '1' represents news content, and the second '1' signifies the first experiment), a shift of x₂₁% in videos, and x₃₁% in user-generated posts. Similarly, subsequent experiments might show variations like x₁₂%, x₂₂%, x₃₂%, and so forth. Here, x can be negative as well.
Subsequently, a straightforward linear regression is performed on the aggregated data at the percentage per treatment level. The regression equation is:
where the coefficients (coeff ₖ) are determined as the elasticities representing the impact of increasing or decreasing the distribution of specific content type xₖ in the experiments.
Interpreting the coefficients
If a coefficient (coeff ₖ) is statistically significant and positive, it indicates that showcasing more of that specific content type to users (while keeping all other content types constant) would likely lead to a positive impact on user value.
For example, suppose we're analyzing the impact of increasing video content (let's call it x₁) in our experiments. The coefficient (coeff₁) associated with x₁ indicates how much the increase in videos affects the observed outcomes. If coeff₁ is positive and significant, it implies that showcasing more videos might positively impact M-1 retention, encouraging users to stay engaged or active over consecutive months.
Additional Pointers
For a more resilient analysis, it's essential to consider and apply these outlined pointers.
The ideal scenario necessitates as many experiments (n) as independent variables (k), ensuring accurate estimations; thus, k should be considerably lesser than n.
To generate new experiments for analysis, diverse treatment strengths, such as altering content distribution by varying percentages (e.g. in one experiment we increase the distribution of meme content by 10%, in another by 20%)), or conducting experiments at different times, prove beneficial.
They noted that conducting a smaller number of clear experiments, involving changes in all content types in at least one trial, works effectively.
Starting with a simple linear regression serves as a solid starting point before delving into more complex ML models.
The linear regression model mentioned earlier assumes constant elasticities. This means that a 1% increase in viewing a specific content type results in a y% change in the outcome variable. It's assumed that a 2% increase in viewing leads to a 2 * y% change. This constant elasticity has been observed at Facebook, scaling consistently across users with varying activity levels.
In summary, Meta's methodology for optimising the Facebook Feed is a systematic process based on comprehensive experimentation, data analysis, and statistical modelling. By prioritising long-term user satisfaction over immediate engagement metrics, they conduct numerous experiments, utilising A/B testing and regression analyses to determine the impact of different content types on user value, often measured by metrics like M-1 retention. Their approach involves examining content shifts, interpreting coefficients from linear regression models, and ensuring robust experiment designs with various treatment strengths. Ultimately, this data-driven approach helps Meta tailor and refine the Facebook Feed to better match users' preferences, creating a more engaging and satisfying experience for everyone.