When User-Engagement ≠ User-Value

Raw text

Transcribed text as recorded, including headers and formatting fragments.

Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 1 of 16 AMENDED Exhibit 139 PLAINTIFFS' OMNIBUS OPPOSITION TO DEFENDANTS' MOTIONS FOR SUMMARY JUDGMENT Case No.: 4:22-md-03047-YGR MDL No. 3047 In Re: Social Media Adolescent Addiction/Personal Injury Products Liability Litigation ForkpX en-use (10) Work Case 4:22-md-03047-YGR engagement-user-value/962729940736135/ Notes Document 2756-59 Filed 02/20/26 Page 2 of 16 (Meta) Zuckerberg Exhibit 15 3.27.25 mlr HOW RUDE! exhibitsticker.com When User-Engagement + User-Value TOM CUNNINGHAM THURSDAY, DECEMBER 10, 2020 READING TIME 11 MINUTES (originally written in 2019, published in Dec 2020). Three broad reasons why increasing user-engagement might not increase user- value: • First, when the product stretches value out – i.e., it spreads value thinly, to maximize engagement. • Second, when the product is misleading – i.e., we make promises about value without fulfilling them. • Third, when users' choices don't represent their own be when they have self-control problems Chats MacBook Pro Haugen_00010114 + Writ Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 eople Tab Wind Page 3 of 16 101 4) Workp (10)Wx (11) 2020-0X Log in to War X (10) Workx (10) Come + /when-user-engagement-user-value/962729940736135/ ☆ Notes • Third, when users' choices don't represent their own best interests - e.g. when they have self-control problems. I think that all of these are important issues for our products. Q: How to make product decisions that maximize user-value instead of user- engagement? I don't have a silver bullet, & just have a few observations. Choosing engagement metrics carefully can help - e.g., maximizing sessions, instead of time-spent, has some positives for the "stretching" problem, although it also may have negatives with respect to self-control issues. • Monitoring sentiment metrics can help - but they're often too noisy to help. • A strong quality culture probably helps. Background 1. There are a lot of critiques of social media that we maximize engagement at the expense of value.. 2. We are providing value. People spend a huge share of their time on Facebook apps - something like 1% of worldwide time-spent - it would be surprising if we weren't delivering some positive value. When you ask people to put a monetary value on Facebook it's high: the average American says it would take $50-100 before being willing to give up Facebook for a month. 3. There are reasons to be concerned. Most importantly the relationship between usage of Facebook and psychological wellbeing is pretty murky: higher FB use is associated with worse wellbeing, in the cross-section (though not in the time-series), and in a recent experiment they found that a 1-month break from Chats MacBook Pro + Write Note Haugen_00010115 Case 4:22-md-03047-YGR Document Document 2756-59 215 People Tab Window Help 20+) Workpl X Filed 02/20/26 Page 4 B @(10) Workplac X | (11) 2020-08 × @ Log in to Work X @ (10) Workplac X (10) Commun X + m/when-user-engagement-user-value/962729940736135/ ☆ T Notes 3. There are reasons to be concerned. Most importantly the relationship between usage of Facebook and psychological wellbeing is pretty murky: higher FB use is associated with worse wellbeing, in the cross-section (though not in the time-series), and in a recent experiment they found that a 1-month break from using Facebook caused a small but significant increase in self-reported psychological wellbeing. 4. Why would engagement and value diverge? Basic economic reasoning would imply that people engage with a product to the degree that they get value from it, so engagement and value should be closely connected. I think this is broadly true - but there are a few wedges where value and engagement can diverge. (1) Stretching Out Value (AKA Spence distortion) Spreading the jam thin. Suppose you have a fixed amount of value to offer but you're trying to increase engagement, e.g. time-spent, then you can do this by spreading the value out: cut the value up into chunks, and put some filler in between. A couple of economic papers in the 1970s by Michael Spence formalized this point: if you're a firm making choices about the design of your product then your incentive is to maximize the quantity of demand - i.e., the number of people who want it, or the time- spent using your product - rather than the quality of demand, i.e. the total value delivered. (More precisely: your incentives are around increasing the marginal value, not the total value). Spence applied this reasoning to television, comparing cable TV to broadcast TV. Cable TV is funded by a fixed fee, so they have an incentive to maximize overall value. Broadcast TV is funded by advertising, so they want to maximize time-spent, AKA engagement. For this reason there's an incentive for broadcast TV to stretch out their value in a couple of ways: (1) if they can make a show more broadly appealing Chats MacBook Pro + Write Note 6 Haugen_00010116 People Tab Window Help 20+) Workpl. X Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 (10) Workpla: X (11) 2020-08 X Log in to Won X (10) Workplac X m/when-user-engagement-user-value/962729940736135/ Notes Page 5 of 16 (10) Commun X ☆ Spence applied this reasoning to television, comparing cable TV to broadcast TV. Cable TV is funded by a fixed fee, so they have an incentive to maximize overall value. Broadcast TV is funded by advertising, so they want to maximize time-spent, AKA engagement. For this reason there's an incentive for broadcast TV to stretch out their value in a couple of ways: (1) if they can make a show more broadly appealing, at the expense of value to the core fans, that is always in their interest; (2) if they can artificially increase viewing time on a show they'll do that - by having constant recaps, and padding, and cliffhangers. Stretching value occurs in our products in a few ways. 1. We limit the amount of information in notifications because we want people to come onto the site. A few years ago we stopped sending out emails telling you what happened - e.g. telling you what your friend did - instead we just say "someone comment on your post," in the hope that you'll-click through. This is a clear value-engagement tradeoff. We see similar effects on push notifications about friend activity: when we send fewer notifications, that increases engagement because people come to the site to check what's happening. 2. Lots of content on News Feed withholds information. With titles like "you'll never guess...", "one amazing trick...," they try to get you to click through, even though they could've easily told you the information up-front. We currently have a ranking penalty for egregrious cases, but you can still see the effects of this incentive everywhere, even high-quality newspapers craft their headlines to leave out some tantalizing detail. 3. Maximizing for time-spent incentivizes bad ranking. If we evaluated ranking by the effects on time-spent then it would not be optimal to rank just by expected value instead we would be better off by adding some randomness or filler, so that the value is stretched out. It's hard to tell how important this is for ranking in practice, it's a subtle thing to measure and ranking is a black-box, but I'd guess it has a significant influence. 4. More friction for high-value things. Suppose that someone Chats MacBook Pro + Write 6 Haugen_00010117 Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 6 of eople Tab Window Help +) Workpi. X (10) Workplac X (11) 2020-08 X Log in to Wor X (10) Workpla: X (10) Commun X + when-user-engagement-user-value/962729940736135/ ☆ Notes 3. Maximizing for time-spent incentivizes bad ranking. If we evaluated ranking by the effects on time-spent then it would not be optimal to rank just by expected value instead we would be better off by adding some randomness or filler, so that the value is stretched out. It's hard to tell how important this is for ranking in practice, it's a subtle thing to measure and ranking is a black-box, but I'd guess it has a significant influence. 4. More friction for high-value things. Suppose that someone really wants to do something - e.g. there's a video that they really want to watch - then any feature or experiment can add friction without having much effect on engagement - e.g. we can add more ads, interstitials, promotions. Again, these types of things will hurt consumer value without having a proportionate impact on engagement. (2) Misleading Users A second reason for divergence between engagement & value is misleading experiences. If you give the impression to users that they're going to get some value by engaging with your product, but you don't actually give users that value, that will introduce a discrepancy between engagement & value. This happens with Feed links all the time. Headlines will say misleading things to get you to click, subtly exaggerate. We have a classifier and demotion for exaggerating headlines, but it's a very crude solution. If we blindly followed AB tests to maximize engagement, we'd lie to users all the time. The simplest way to maximize engagement is just to lie to users, e.g. by saying "click this link for guaranteed long battery life", or "accept our terms and we promise not to track you." If an AB test has a surprisingly positive effect on engagement you should double-check that the feature isn't somehow inadvertantly misleading users. Why doesn't this process self-correct? You'd expect that consu Chats MacBook Pro F + Write Note 6 Haugen_00010118 ople Tab Window Help Case 4:22-md-03047-YGR (11) 2020-00 )Workple X (10) Workpl when-user-engagement-use Notes X -value/962729940736135/ Document 2756-59 Filed 02/20/26 Log to Wo in (10) Workp Page 7 of 160 10) Commer X Why doesn't this process self-correct? You'd expect that consumers will learn when firms mislead or exaggerate, and adjust their expectations, giving firms an incentive to tell the truth. This doesn't work perfectly in practice for a couple of reasons: (1) the cost of disappointed expectations is shared – when users have a hard time keeping track of who's responsible for the content they're seeing they'll blame all publishers, and all app developers, for disappointed expectations. This weakens the incentive to tell the truth, because the costs are borne by other companies. (2) many features are evaluated just on short-runengagement if we mislead users it'll surely cause a lack of trust down the line, but that can be slow and Jumulative, and it's very hard to detect with just short-run experiments. (3) Self-Control Problems Finally there are sometimes discrepancies between what people choose and what gives them the most value. The classical economic point of view is that a person's choice perfectly represents their preference – they're the same thing. However there are a bunch of cases where that seems overly simplistic – e.g. when people have self-control problems, or cognitive problems in processing information. There are reasons to worry about self-control and use of our products. I think the biggest problems people have are: (1) compulsive use, to fill gaps in their time, and they end up spending more time than they'd like, instead of spending time with their families, or studying; (2) unhealthily fixating on other people – most prototypically, compulsively looking up the profile of your ex-partner. A quick rundown of evidence: 1. Higher Facebook use is correlated with worse psychological states (wellbeing, loneliness, etc.), although changes in Facebook use are uncorrelated with changes in wellbeing measures. [8,5] Chats MacBook Pro * Haugen_00010119 Case 4:22-md-03047-YGR People Tab Window Help Document 2756-59 Filed 02/20/26 Page 8 of B 20+) Workpl X (10) Workpl X (11) 2020 0 X Log in to Wc X (10 Workpl X (10) Commun x + m/when-user engagement-user-value/962729940736135/ ☆ Notes 1. Higher Facebook use is correlated with worse psychological states (wellbeing, loneliness, etc.), although changes in Facebook use are uncorrelated with changes in wellbeing measures. [8,5] 2. An experiment found that a 1-month break from Facebook improved self- reported wellbeing. [6] 3. A large fraction of users struggle with their Facebook/Instagram use. Among US teens "84% had taken a break from social media, and 74% have taken a break from ►Instagram." [2] “More than a third of the survey respondents feel they have spent more time than they would have liked on Instagram. About one in five respondents think they have a problem controlling their time." [5] 4. A significant minority report serious difficulties. 3.1% of Facebook users in the U.S. report “serious problems with sleep, work, or relationships that they attribute to Facebook AND concerns or preoccupations about how they use Facebook." [1] What do Do? Despite lots of work I think it's still an open problem, of how to make sure that maximizing user-engagement doesn't harm user-value. I'm planning to spend some more time thinking about this, in the meantime here are a few observations: 1. Choosing engagement metrics carefully can help. over the last couple of years News Feed slowly switched from maximizing time-spent towards maximizing sessions. This should be good from the point of view of the Spence distortion: the frequency of sessions should be roughly proportional to the total expected value, so there's less incentive to stretch value out to maximize time-spent. However this could be bad from the perspective of self-control problems: we actually find that the frequency of sessions is a strong predictor of problematic use, and our own internal reseomobore rodontlu recommended we should “help people consolidate their use of Fa MacBook Pro Chats F + Write Note 6 Haugen_00010120 People Tab WiCase 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 9 of 16 20+) Workpix Ⓒ(10) Workplac X (11) 2020-08 X Log in to Wor X (10) Workplac X (10) Common x + /when-user-engagement-user-value/962729940736135/ ☆ OF Notes 1. Choosing engagement metrics carefully can help. - over the last couple of years News Feed slowly switched from maximizing time-spent towards maximizing sessions. This should be good from the point of view of the Spence distortion: the frequency of sessions should be roughly proportional to the total expected value, so there's less incentive to stretch value out to maximize time-spent. However this could be bad from the perspective of self-control problems: we actually find that the frequency of sessions is a strong predictor of problematic use, and our own internal researchers recently recommended we should "help people consolidate their use of Facebook into fewer sessions." [1] 2. Monitoring sentiment metrics can help. - e.g. CAU, WYT, etc., but my impression is that they're often too noisy to give guidance to everyday product decisions. 3. A strong quality culture probably helps. – I sometimes wonder what the alternative is to making decisions based on AB-tests of engagement metrics, and I scratch my head, then I remember that just over the road WhatsApp built a 1Bn user product without running any tests, they made all their product decisions just based on their perceptions of user quality. I don't feel qualified to judge on whether the outcome is better, but it's certainly worth remembering that there is an alternative. (Of course WhatsApp cares about growth and engagement, but the fact they didnt' run experiments until recently implies that they don't think short-run AB-test measures were very valuable measures of long-run value). References [1] Cheng, Burke, Davis, "Understanding Perceptions of Problematic Facebook Use", https://www.dropbox.com/s/bph75wxf1u27hnr/problematic_use_2019-01-07_final.pdf? dl=0 MacBook Pro Chats + Write Note 6 Haugen_00010121 Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 10 of 16 Haugen_00010122 Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 11 of 16 Haugen_00010123 Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 12 of 16 Haugen_00010124 Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 13 of 16 Haugen_00010125 Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 14 of 16 Haugen_00010126 Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 15 of 16 Haugen_00010127 Case 4:22-md-03047-YGR Document 2756-59 Filed 02/20/26 Page 16 of 16