Email: Re: (Decision needed) Revert jewel filtering on US iOS decision

Raw text

Transcribed text as recorded, including headers and formatting fragments.

Message From: Andrew Bocking [/O=THEFACEBOOK/OU=EXTERNAL (FYDIBOHF25SPDLT)/CN=RECIPIENTS/CN= Sent: 1/9/2018 4:10:47 AM To: Javier Olivan @fb.com]; Naomi Gleit @fb.com]; Alex Schultz @fb.com] CC: Guillermo Spiller @fb.com]; Denise Moreno @fb.com]; Steve @fb.com]; Robert d'Apice @fb.com]; Brady Lauback @fb.com] Subject: FW: (Decision needed) Revert jewel filtering on US iOS decision Attachments: Re_Understanding Growth Metrics.msg; image001.png; image002.png FYI on my last note to Adam incl. the slides/note to Mark just now pushing for his support. Will let you know how he responds, and would appreciate if you can all help push when you see the chance too including in the review tomorrow. From: Andrew Bocking @fb.com> Date: Monday, January 8, 2018 at 8:03 PM To: Adam Mosseri < @fb.com>, Guillermo Spiller @fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision Hey Adam Yes... I understand that NF/Notifications serve many interests, but we got a very clear and strong message from Mark that DAP (and specifically US DAP) is extremely important and we must change the trajectory from the negative one it's on now. It's an externally reported metric and therefore needs the right level of priority. I think it would trump video watch time and many other metrics in priority given this? Make sense? Attached is the final slides + next steps from today's 3pm. Mark wants us to 'fix' US DAP ASAP is the main tl;dr... We will need everyone's support to do so, even if it's just 150K US DAP here and there... can you help? Javi definitely wants the Notification change reverted, as well as pushing hard on any intentional changes (bucket #2 in the slides) that may be hurting. I know you guys are being pushed really hard on Sharing and I'm 100% on that same page. I think this is more around making sure our other work (e.g. quality) is also focused on conclusively not hurting (and ideally improving) DAP. Andrew From: Adam Mosseri @fb.com> Date: Monday, January 8, 2018 at 5:59 PM To: Andrew Bocking < @fb.com>, Guillermo Spiller < @fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision I get that Growth is primarily concerned with US DAP right now, not on NF and Notifications we juggle the interests and metrics of lots of teams. Sharing is important. So is planned video watch time. So is friending. So are meaningful social interactions. My point is if every change we make has to be neutral or positive on every metric we run the risk of paralysis. Sent from my phone From: Andrew Bocking Sent: Monday, January 8, 2018 2:33:24 PM To: Adam Mosseri; Guillermo Spiller Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) PLTF EX PLTF EX-00462 ΜΕΤΑ3047MDL-004-00017391 METANMAG-002-00209728 PLTF EX-00462_0001 Hey Adam Answered the Q about 3pm attendance on Messenger so won't here. On the first point - I think we're not saying 'any metric' we are specifically saying Visitation (DAP), and for that matter Friending, are so related to visitation AND engagement we should ALL care about these. Really layman simplification of the 'fly wheel'.... Make sense? DAP MAP Engagement Critical mass Rising adoption Haven't tried From: Adam Mosseri DAP Declining activity @fb.com> Repeat cycle MAP | on next thing Don't use Date: Monday, January 8, 2018 at 1:46 PM To: Andrew Bocking < @fb.com>, Guillermo Spiller @fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision Thanks for the context. It makes a lot of sense to me that we cares lot about US DAP, but I worry about becoming myopic about any metric. If we're unwilling to take any hit on any key metric, and we keep adding key metrics, my take is we'll likely slow down our work so badly that we'll be in much bigger trouble. On this it's important that we don't position this is just about some subjective notion of quality, but we see quantitative evidence that the notifications channel is too noisy and real engagement wins from the work. Who's in the 3pm meeting? Sent from my phone From: Andrew Bocking Sent: Monday, January 8, 2018 10:36:11 AM To: Adam Mosseri; Guillermo Spiller Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision Hey Adam To add - just got clear input from Naomi that US DAP is a bigger concern for Mark right now than user experience. We have been pulled to see him this afternoon now at 3pm about this. Andrew HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017392 METANMAG-002-00209729 PLTF EX-00462_0002 From: Andrew Bocking @fb.com> Date: Monday, January 8, 2018 at 9:57 AM To: Adam Mosseri < @fb.com>, Guillermo Spiller @fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision Hey Adam Thanks for reaching out, and welcome back - I hope your parental leave was awesome! I think the important context here is the broader state of US DAP, where we just went negative (by ~700K) in Q4 Q/Q for the first time. US MAP ended -2.64M behind goal (only 50% of our H2 expectations), and US DAP ended -3.14M behind goal (only 20% of our H2 expectations). Notably, final Q4 Facebook DAU totals for the US & Canada region declined compared to Q3 2017, from 184.7M down to 184.0M (-0.7M, -0.4% Q/Q; +3.7M, +0.1% Υ/Υ). This is the first time we've had this sequential Q/Q metric be negative. Q4 2016 was +2.2M Q/Q for the same metric, and Q3 2017 was +1.7M Q/Q. We've done a lot to try to understand what's going on and are still aggressively looking for other factors/causes and have a dedicated taskforce/war room spun up now. For more context, below is the excerpt from our H1/H2 doc for the U.S. you likely saw in the pre-read if you've read it already. Our over-arching concern is that we currently have many macro factors and every headwind we create for ourselves (even if relatively small by itself) all add up and growth is on the margin, especially in more saturated growth markets like the U.S. Given this backdrop, we believe the more our engagement work also focuses explicitly on improving DAP (& DAP/MAP) the better as our work to improve quality (e.g. notifications) should be positive for visitation. Basically, we feel everyone who visits Facebook finds value and we should keep working to get all users visiting even marginal users. Instead of stopping the work to get all users to the site we should focus on helping them become more engaged over time. That said - we definitely agree the current notification experience is not the best, but we think we can make the experience better without losing DAP (even the marginal ones). Totally agree this is controversial to have teams focus on DAP as well as other engagement metrics (e.g. MSI), and this is why it's tabled as our main discussion question for H1/H2. NOTE: Beyond the summary below, we are actively investigating a Friending trend break/decline (found yesterday) that looks like it started around the same time as the US DAP trend break. We hope to know more about this in the next 24hrs. Thoughts? Does this make sense? Thanks! Andrew U.S.: In H2, we ended significantly behind expectations on MAP and especially DAP where DAP declined Q/Q in Q4 for the first time. After accelerating growth in late 2015 and early 2016, a deceleration began in late 2016 and has continued through H2 and especially in Q4. The DAP/MAP ratio has also been declining or plateauing in older cohorts which is very troubling. We believe this decline is due to the following primary causes. These causes are very hard to quantify outside our own product changes. Even then the impact is difficult to accurately and fully measure with holdouts given wide error bars and potential network effects. HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017393 METANMAG-002-00209730 PLTF EX-00462_0003 1. Engagement Metric Decline: Declines in sharing, reshares and comments correlate with the deceleration. We still need to better understand the interplay with DAP and relationship with other identified causes (e.g. are iOS, teens, and/or competition the cause of metric declines), but an explicit focus on DAP in our engagement efforts should help. 2. Intentional Product Changes: Stories/Camera and quality-related product changes including notification filtering, video recommendations, and feed ranking (est. -1.7M to +0.5M DAP in total). 3. Competition/Family: Continued strength, especially on iOS, of Instagram (e.g. ranked feed hurt DAP by ~300K), iMessage, Snapchat, and Youtube. 4. iOS Ecosystem: More people are shifting to iOS where we have tougher competition and teens do worse. iOS 11 also introduced new browser cookie/login issues with ITP (est. -530K MAP / -110K DAP) and performance issues (est. -200K DAP). 12.5 M 10 M 7.5 M 5 M 2.5 M US DAP Y/Y Note: other factors like country attribution can add marginal noise to trends. 0 Jan '13 Jul '13 Jan '14 2013 metric IG Feed Ranking Engagement metric decline begins -2016 election day Acceleration in engagement metric decline Stories/Camera iOS11 Quality-driven product changes begin (i.e. notification filtering) vs. 2016 election day Jul '14 Jan '15 2014 metric Jul '15 2015 metric Jan '16 2016 metric Jul '16 Jan '17 Jul '17 Jan '18 2017 metric In H1, we have setup a dedicated team to define and execute on a strategy for these negative trends and will partner with Core App. From: Adam Mosseri @fb.com> Date: Monday, January 8, 2018 at 9:18 AM To: Guillermo Spiller @fb.com>, Andrew Bocking < @fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision Just to you two. I've been out for a while and I'm just ramping up so I may be lacking some context, but wanted to take on why this was contentious. I understand US DAP is very sensitive, but 130K I believe is just single digit basis points of that total, and I worry about how we can continue to make meaningful progress if we're not willing to take such small trade-offs (both for News Feed and Notifications) or more specifically I don't see how we move as fast as we need to if everything has to always be neutral or up. You both have consistently been great partners, and I really respect your opinions, so wanted to get your takes. HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017394 METANMAG-002-00209731 PLTF EX-00462_0004 Thanks in advance, Adam From: Guillermo Spiller @fb.com> Date: Thursday, January 4, 2018 at 5:21 PM To: Lars Backstrom < @fb.com>, Chris Cox < l@fb.com>, Javier Olivan < @fb.com>, Alex Schultz @fb.com>, Mark @fb.com> Cc: Naomi Gleit < John Hegeman Moreno < @fb.com>, Andrew Bocking @fb.com>, Adam Mosseri @fb.com>, @fb.com>, Ronan Bradley @fb.com>, Dan @fb.com>, Denise @fb.com>, Brian @fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision (+Denise and Brian) Hi all, Below I'm summarizing next steps based on the thread and defining owners for them. Let me know if I'm missing anything, otherwise I'll kick-start independent threads with the POCs to coordinate the following: 1. Deeper analysis: As few people suggested, let's do a deeper analysis to be able to answer whether this change is good for all users (most notably marginal users). This analysis should work as a guide for the things to look at on future changes. POC: Ronan and Dan from Core App, Denise and Guillermo from Growth 2. How to work going forward: #1 should inform how to analyze changes and make sure we build a good user experience for everyone. In addition, we should continue iterating together on how to think about the notifications system and how the teams should work together (goals, process, monitoring, etc.). POC: Mark from Core App, Denise and Guillermo from Growth Thanks, Guillermo From: Lars Backstrom < @fb.com> Date: Sunday, December 24, 2017 at 11:54 AM To: Chris Cox < @fb.com>, Alex Schultz < @fb.com>, Javier Olivan < Spiller < Cc: Naomi Gleit John Hegeman @fb.com>, Mark @fb.com>, Guillermo @fb.com> @fb.com>, Andrew Bocking < @fb.com>, Ronan Bradley @fb.com>, Adam Mosseri < @fb.com>, Dan @fb.com>, @fb.com> Subject: RE: (Decision needed) Revert jewel filtering on US iOS decision Alex, I agree that we should continue to get more sophisticated in how we target and filter notifications and I think that together growth and news feed have plans for differentiating the experience further to address some of the issues you are flagging in H1. Given that we've already made good progress mitigating some of the drop by reducing filtering for some of the marginal users you mention, I'm fairly confident that we can fix forward in H1 and continue to achieve the current level of reduction to the overall notification level with a smaller and smaller cost to DAP. To me, the main reasons to fix forward boil down to two points: 1. The current downside is relatively small and something I think we can tolerate for a while. While 130K US DAP is important, it is something we can live with and iterate on. Also, as far as we can tell US iPhone WAP seems HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017395 METANMAG-002-00209732 PLTF EX-00462_0005 neutral (https://fburl.com/2p2mv7u4: 0.0% +/- 0.03%), so it seems likely that we aren't losing many people forever, and so our future improvements can continue to close the DAP gap to the extent that we find it is due to things like marginal users not getting their first like notification, or other "bugs" in our current optimization. 2. We will make more progress improving every aspect of the system in H1 by fixing forward rather than reverting. It is always easier to run many parallel incremental experiments and ship the ones that are good rather than trying to bundle lots of changes together and ship a big package. Even if one thinks that the current tradeoff is not net-positive (though I would argue that it is net-positive overall), if we are optimizing for the end of H1 rather than the short term, we'll make more progress and end up better off by iterating forward. Do we feel like we have the right metrics in place to be able to spot concerns like the ones that Alex raises? We should be working together on a common set of visitation and engagement goals, with quality considerations taken into account. Do we have a shared viewpoint on what all the metrics and goals we should be tracking in H1 are? I think it would be very useful to circulate that to this group to make sure we're all on the same page about what we are going to try to achieve in H1 from both a quantitative and qualitative point of view, and resolve any differences of opinion on what our H1 goals should be. -Lars From: Chris Cox Sent: Sunday, December 24, 2017 7:59 AM To: Alex Schultz < @fb.com>; Mark l@fb.com>; Guillermo Spiller < @fb.com>; Javier Olivan @fb.com> Cc: Naomi Gleit < @fb.com>; Andrew Bocking < @fb.com>; Adam Mosseri @fb.com>; John Hegeman < @fb.com>; Lars Backstrom @fb.com>; Ronan Bradley < @fb.com>; Dan <@fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision Adding Ronan and Dan Can you guys dig into whether this is hurting marginal users as Alex suggests? I'd like to know for certain whether or not this is churning out marginal users for whom we don't have a meaningful volume of notifications. From: Chris Cox @fb.com> Sent: Saturday, December 23, 2017 1:22 PM Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision To: Alex Schultz @fb.com>, Mark @fb.com>, Guillermo Spiller @fb.com>, Javier Olivan < @fb.com> Cc: Naomi Gleit @fb.com>, Andrew Bocking @fb.com>, Adam Mosseri @fb.com>, John Hegeman @fb.com>, Lars Backstrom @fb.com> These are good questions and we should look into them. HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017396 METANMAG-002-00209733 PLTF EX-00462_0006 With that said, as a principle I want to empower the notifications team to make these decisions based on what they believe is best for the overall UEX. Let's operate with that as the principle, and make sure we're paying attention to the question Alex raises as we go. From: Alex Schultz < @fb.com> Sent: Saturday, December 23, 2017 12:39 AM Subject: RE: (Decision needed) Revert jewel filtering on US iOS decision To: Mark Olivan < @fb.com>, Guillermo Spiller < @fb.com> @fb.com>, Chris Cox @fb.com>, Javier Cc: Naomi Gleit < @fb.com>, Andrew Bocking John Hegeman @fb.com>, Lars Backstrom @fb.com>, Adam Mosseri < @fb.com>, @fb.com> Hey team, I worry a lot that when we do something we think is good but a metric goes down it signals something we don't understand. So when I see a metric is down, esp. one like are people stopping using facebook or incrementally not using facebook, I think we've probably done something bad somewhere. It's pretty final to lose people from the site. I spent a bunch of time reading this note this weekhttps://fb.facebook.com/notes/jerrod-lowmaster/how-the-other-half- lives/448767755525511/?qid=6502392156458899727&mf story key=-2370534856086033192 where another analytics person rediscovered that averages are the wrong way to look at our user base and how our way of measuring averages ignores the marginal user and biases us to power users as does our personal experience. It's worth reading and thinking about. So many users are on the margin, so many users are so different to us that what we think is an awesome change for us isn't for them. This video is a great example:https://www.youtube.com/watch?v=zG1796352RI where if you have 8Million followers like notifications suck and all of us are people who have experiences where our jewel is lighting up all the time (if not this badly). The flip of that is that if it's your first like it can be one of the most meaningful notifications on the platform you ever receive. Fundamentally I believe that we have abused the notifications channel as a company. The biggest violators for me are things like app invites and group invites and historically we weren't able to remove them when my team pushed on removing them because they hurt metrics so I understand the feeling on not stopping for a metric BUT I believe metrics are a warning signal, not all of our users look like us and usage metrics (time, visitation, active users) are qualitatively v. different than using groups or using apps on our platform. Are we really sure we've thought about the marginal user well enough in this work? Could these metrics declines be a flag there is something we aren't understanding? Alex From: Mark Sent: Thursday, December 21, 2017 10:30 PM To: Guillermo Spiller @fb.com>; Chris Cox Cc: Naomi Gleit < @fb.com>; Andrew Bocking < Mosseri < @fb.com>; John Hegeman < @fb.com>; Javier Olivan @fb.com> @fb.com>; Alex Schultz @fb.com>; Adam @fb.com>; Lars Backstrom < @fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision One quick clarification from News Feed side: HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017397 METANMAG-002-00209734 PLTF EX-00462_0007 News Feed team does believe fewer notifications is better because in other platforms and generally around the world, engagement downstream has increased (e.g., comments, rsvps, watch time, et.al.). It makes sense on the surface: Send fewer things people don't seem to want, they'll be better able to discover the things they do want, and act on them. US IOS is behaving differently, and we don't know why yet, but it doesn't behave worse. Also, it's not clear that anything is broken, but we all agree there's room for improvement: Filtering will get smarter, we can better understand why US IOS is behaving differently, and we can account for incremental visitation in the future. Given we don't believe the DAP decline is symptomatic we've made the experience worse in some way, we will not revert at this point. We'll continue to understand the US IOS differences and report back if there's some material finding. From: Guillermo Spiller @fb.com> Date: Thursday, December 21, 2017 at 1:42 PM To: Chris Cox < @fb.com>, Javier Olivan < @fb.com> Cc: Naomi Gleit @fb.com>, Andrew Bocking @fb.com>, Alex Schultz < Adam Mosseri < @fb.com>, Mark l@fb.com>, John Hegeman @fb.com>, @fb.com>, Lars Backstrom <@fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision Thanks Chris. Addressing your points: I think it's hard to know whether fewer notifications is better for people. But we think that we could have broken something because: 1. The filtering doesn't take into account visitation value, it only considers downstream actions (comments, rsvp, live video watch time, wall post, friend requests) 2. We see positive engagement (u2u comments, live video watch time) impact on Android but not on iOS; which could mean that iOS filtering isn't as good 3. The filtering doesn't take into account incremental value to people. Meaning, whether the action that the notification incentivizes would have happened without the notification; so we could be filtering the wrong notifications We agree on improving filtering but given the 3 points above and the negative DAP impact, we think we should revert and iterate. (We are all committed to iterate but it might take a few months to get it right so the short-term revert decision is important) Guillermo From: Chris Cox @fb.com> Date: Thursday, December 21, 2017 at 11:42 AM To: Guillermo Spiller < @fb.com>, Javier Olivan @fb.com> Cc: Naomi Gleit @fb.com>, Andrew Bocking @fb.com>, Alex Schultz Adam Mosseri < @fb.com>, Mark l@fb.com>, John Hegeman < @fb.com>, @fb.com>, Lars Backstrom < @fb.com> Subject: Re: (Decision needed) Revert jewel filtering on US iOS decision If we think that the filtered experience is better for people, I feel strongly that we shouldn't revert this because a metric is down. The heart of the matter is that we need to get better at making the harder decisions where the metric isn't the main decision criteria: the experience is. HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017398 METANMAG-002-00209735 PLTF EX-00462_0008 If we think the DAP declines is symptomatic of the experience being *worse* in some way, than I'm open to slowing things down. I'd ask the team to spend time on understanding the decline from that perspective. If we then feel like we've broken something or deteriorated it here, we can consider a more phased rollout. To: Javier Olivan < Cc: Naomi Gleit Date: Thursday, December 21, 2017 at 11:36 AM @fb.com>, Chris Cox < @fb.com>, Andrew Bocking @fb.com> @fb.com>, Alex Schultz < Adam Mosseri < @fb.com>, Mark l@fb.com>, John Hegeman @fb.com>, @fb.com>, Lars Backstrom < @fb.com> Subject: (Decision needed) Revert jewel filtering on US iOS decision Chris & Javi, We're asking for your help to decide on a short-term trade-off between US DAP and notifications experience. We need to decide whether to revert the notifications filtering change inUS IOS. As part of the Feed Notifications Quality sprint (Stop The Line initiative), News Feed notifications team has been filtering notifications by using a threshold model that optimizes for clicks and downstream actions. This work has increased engagement globally (but not in US iOS) as well as a reduction in notifications sent and negative feedback. But it hurts US DAP, negative impact coming from iOS (details below). We have 2 options to move forward: Option 1 (Core Growth recommends): Revert the Filtering Change Temporarily on iOS in the US. We will work together quickly on a model for notifications filtering that considers visitation prediction that we hope to launch by end of Q1. This would revert up to 70M people from the filtered experience (we don't understand downstream impact of such change) to benefit DAP (+130K, which is 2% of expected US growth this half) in the US until it launches. Option 2 (Core App recommends): Choose not to revert the change on iOS in the US. The News Feed team is committed to fixing forward and continuing to help remediate DAP with better filtering and more precise targeting of at- risk people. Core App believes the user experience impact is not worth the visitation tradeoff. Ultimately, the short-term trade-off centers around: A. Preserving a better notifications experience for people. While it looks like the downstream effects appear to be neutral (e.g., no change to comments), Core App thinks that a potential rollback is negative from a user experience standpoint. Filtering has eliminated 240 million jewel notifications/day (24%) for US iOS people. We estimate removing filtering in US iOS will increase a P90 person's daily notifications from 45 to 55/day, incremental push notifications will increase 306 million/day (17%) and incremental email will increase 130 million/day (38%) while we'll get an incremental 2.4 million negative feedback/day (24% more people hiding notifications). VS B. Recovering US DAP impact. We are expected to end the half 2.7M behind on US DAP, -41% of H2 goal; and trending to negative Q/Q US DAP growth. Reverting will increase iOS US DAP by 130k (2% of the expected growth for US in the half) and if we revert today, we would get +40k US DAP on reported Q4 numbers (given 30 days average), getting us from expected -200k to -160k Q/Q. You can see more details on the tradeoff and context here: https://fb.quip.com/c3adAOpZRzBy Do you think we should revert this change (option 1), or choose not to revert (option 2)? HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017399 METANMAG-002-00209736 PLTF EX-00462_0009 Thanks, Guillermo HIGHLY CONFIDENTIAL (COMPETITOR) HIGHLY CONFIDENTIAL (ATTORNEY'S EYES ONLY) ΜΕΤΑ3047MDL-004-00017400 METANMAG-002-00209737 PLTF EX-00462_0010