Meta & Children's Online SafetyChat: Meaningful Engagement is a proxy for future retention
Thread View
████ name redacted. Different bars may be the same person or different people.
Tuesday 14 December 2021
06:00████Another example of great data science work:
The Stories ranking model is a multiplicative model. That means we multiply all our predictors together to generate the final ranking score.
To break it down futher, we have a handful of models that try to predict different events, like p(tap) or p(BFF). Each predictive model generates its own score. Those score gets multiplied against the scores from the other predictive models, to generate a final score that we use to rank all the Stories.
https://www.internalfb.com/intern/wiki/Instagram_Home_Ranking/Beginner's_Introduction/Stories_Ranking/1- _Predictive_Model/
This half, we a predictive model to predict Stories replies and a predictive model to predict if a user will send a direct reshare of a story. David observed that p(reply) is highest on stories from someone you actually know in real life. But p(reshare) happens more often on interest content. So, pogs would usually have a high score on one model or the other, but never both. As a result, a low score would be multiplied against a high score, leading to the downranking of stories that have a high probability of starting a conversation. So, we are going to start adding together p(reply) and p(reshare), instead of multiplying them against eah other.
Early tests show that this could end up being one of the most impactful launches of the half, all based on that data science insight that different types of stories drive different types of conversations.
06:00████One of the highest impact DS projects I've ever seen was the development of a ranking optimization metric, what we now call Meaningful Engagement. This work is honestly what sold me on joining this team.
On ecosystems, one of my responsibilities was upleveling the best DS work at IG, and the Meaningful Engagement project was so superior to anything I had ever seen before on any other DS team.
Ranking is a series of functions that try to maximize a metric. Before Meaningful Engagement, ranking optimized for impressions. Engineers would build new iterations of the ranking models, and if impressions went up, we would ship. Impressions is a bad metric for many reasons. We would try to also consider other types of engagement, but the tradeoff conversations were messy and inconsistent. So, the DS team built a composite metric, combining many forms of engagement.
Meaningful Engagement is a proxy for future retention. We run a regression, trained on many types of engagement events to predict future session outcomes. This regression determines the value of engagement we can observe in the short term in our experiments, like story replies or dwell time. We are constantly improving Meaningful Engagement, and the results are astounding. The ranking team has consistently driven +0.1% DAU per half for a year now!! It drives revenue! I've never seen anything like it! This half, most of the feed ranking engineering team spent all their time trying to build new predictors to maximize Meaningful Engagement. As a result, we blew our goals out of the water.
https://fb.workplace.com/notes/1203881303399259/
07:58████yess 1 woulu really like to chat with him actually
07:58████1 can reach out to him as well! and if he comes to the office maybe we can go grab a coffee
07:59████these examples are really really exciting thank you for sending!!! both you & have been really helpful in outlining what good DS work looks like on your team
07:59████thought was really good too from our 30 min convo
08:00████invited me to an experiment review later this week which i will go check out! and then, not sure if there is a team lunch that i could join?
09:24████so glad to hear that! yes, this team is super talented. I'm constantly impressed
09:25████yeah he's great, obviously only hires the best
10:03████I'm setting up lunch for tomorrow, can you make it work?
11:00████yes i can!! 1 m free from 122 so anytime is good
11:48████yay!
16:42████okay here is the low down on lunch: the attendees will be who works on creator feed ranking, won't be in the office tomorrow, but she comes in Trequently.
who you know.
is also in the office frequently, but he is working remote in LA for the rest of the week. Most of the eng team is based in NY but they haven't been back in the office yet. The eng tech lead, is AMAZING, I can't wait for you guys to meet, but he is remote from Chicago. I also invited DE. We ve been sharing DE with other ranking teams, but we just hired our first dedicated Sharing Relevance DE this week, and she'll start in late January @delay
16:48████if you want to join any other meetings this week: there is a cross team ranking understand meeting, a stories xfn team meeting, a meeting with s entire ds team, and a meeting with my ds team @silent