Untitled document discussing Nielsen deactivation study

Raw text

Transcribed text as recorded, including headers and formatting fragments.

Case 4:22-md-03047-YGR Document 2885-28 Filed 03/26/26 Page 1 of 7 AMENDED Exhibit 87 PLAINTIFFS' OMNIBUS OPPOSITION TO DEFENDANTS' MOTIONS FOR SUMMARY JUDGMENT Case No.: 4:22-md-03047-YGR MDL No. 3047 In Re: Social Media Adolescent Addiction/Personal Injury Products Liability Litigation Case 4:22-md-03047-YGR Document 2885-28 Filed 03/26/26 Page 2 of 7 Overview This is a research project designed to be one of our first causal approaches to understand the impact that Facebook has on people's lives. Collaborating with internal experts throughout the company with advisory from external experts, we have designed an experimental deactivation study, in which we will randomly ask some people to stop using Facebook and Instagram for a month (compared to a group who will continue to use as normal), helping us explore the impact that our apps have on polarization, news consumption, well-being, and daily social interactions. We'll partner with Nielsen and use a combination of (a) surveys, (b) log data from Facebook and Instagram, and (c) usage data from smartphones. There have been other recent studies (e.g., [HYPERLINK "https://pacscenter.stanford.edu/wp- content/uploads/2019/05/SSRN-id33086402.pdf" \h ]) with similar goals and methods, however, our study is the first study, to our knowledge, that relies on mobile sensor data as well as FB and IG log data (not just self-reported data); attempts to account for expectation effects, and has amped up other aspects like better measures for well-being, compliance checking, sample demographics, etc. We are currently fielding a 1-week deactivation pilot with Facebook only to runthrough the complexities of the study and work out any unforeseen issues. Internal researchers: (Strategy/Well-being) (Demography & Survey Science) (CDS Economics) Moira Burke (CDS) (CDS) (Civic Integrity) Shilpa Mody (Instagram Well-being) Note: Everyone involved in the project has a PhD and specializes in an area associated with the study (e.g., well-being, politics, economic impact, etc.) Partner: Nielsen. Nielsen adds some external credibility to the study, allows us to run this unbranded, and allows us to analyze the data in a deidentified way (so we never access PII). External researchers who have directly / indirectly advised on various aspects of the study: Robert Kraut (CMU) Andrew Hall (Stanford) Avinash Collis (MIT) Jonathan de Quidt (Stockholm University) Matt Gentzkow (Stanford) Hunt Allcott (NYU) HIGHLY CONFIDENTIAL (COMPETITOR) META3047MDL-019-00095647 Case 4:22-md-03047-YGR Document 2885-28 Filed 03/26/26 Page 3 of 7 How long has this project been in the works? We started scoping out this project in January 2018. We met with Matt Gentzkow in February 2018 and learned about his study. Below is a timeline of their study compared to ours. We thought about partnering with him (or other academics), but [ HYPERLINK "https://fb.quip.com/yX4VAKZIoK1P" \I "ANVACA30YDc" \h ]. Ultimately, we decided to run a pilot version on our own and test the process out, and then collaborate with academics more fully afterwards. Comparing our timelines: Our deactivation study Stanford/NYU deactivation study Planning/Scoping January 2018 Privacy XFN review May 2018 Privacy XFN approval (Alcott et al.) January 2018 N/A Launched Approval 1: December 2018 (then FB study app situation occurred, and put this back under review) Approval 2: June 2019 (then went into contract negotiation with Nielsen) N/A Oct/Nov/Dec 2019(!) (pilot) Sept/Oct/Nov 2018 Published N/A January 2019 (NYTimes) April 2019 (nber) Our biggest privacy XFN challenges: Mobile sensor choice: Our study involves collecting mobile sensor data (because self- reported data is not reliable), but we had to find a mobile sensor data collection app that passed all of our standards. Our originally selected app was put under rigorous review after the FB App Study made it to the press, and we had to find a new app that was not sideloaded and did not collect data using VPN or SSL bump. Finding and vetting the app options took us months. Apple: There was sensitivity with Apple and mobile sensor data collection because of the FB App Study; we could not find a mobile sensor data partner that abided by our standards for mobile data usage, so ultimately we are just studying Android users with mobile sensors (but still surveying iPhone users). HIGHLY CONFIDENTIAL (COMPETITOR) META3047MDL-019-00095648 Case 4:22-md-03047-YGR Document 2885-28 Filed 03/26/26 Page 4 of 7 Teens: Our original study had teens in it, but after the FB App Study made it to the press, we were asked to remove teens. We will keep teens in for Instagram, but not meter them, and we will beef up our consent. Facebook will not study teens as of now. Publication / External communication: There are concerns about publishing our final report, and even talking about this study externally. There were even fears of running this study and what will we do if we find out negative things about Facebook (comparisons to cigarette studies). We had to get this study reviewed all the way up to Nick and Javi. Branding: We have to run this study unbranded because of the impact branding could have on people's perceptions of the study, whether or not they are willing to participate, and what they think/expect to happen. However, we wanted to do this in the most ethical way possible so we disclose Facebook as the funder at the end of the study (and give people a chance to remove their data). Asking about certain topics: We got pushback from both FB and Nielsen's review about certain questions around political viewpoints, political affiliations, and well-being-related topics. Study is complicated: This study is extremely complex, lots of pieces to it, and it's a lot of information to parse. We've had to do lots of 1:1s and group meetings with people to explain details. In the midst of all our planning, the Alcott et al. study came out, which caused us to rethink certain elements of the study (e.g., expectation effects), so we had to evolve our study, which made the privacy XFN piece even more complicated. Here is [ HYPERLINK "https://fb.quip.com/ayptAbnSjvbS" \I "TcDACA3bRyv" \h ] of us outlining all the ways our project changed from when it was first approved. XFN turnover: Natural turnover from people leaving Facebook, going on leave, etc. Our biggest study challenges internally and externally (and our solutions): Challenge Expectation Effects tl;dr In most experiments people are blind to what condition they are in so they can't guess the study goals. In a deactivation study, people will know what condition they are in, and if they have strong expectations they may alter their behavior and/or survey responses. Solutions Reliance on mobile sensor data: Behavior can still be subject to expectation effects, but we think self-reported responses will be more susceptible than behavioral outcomes. Primes: We're attempting to measure expectation effects utilizing primes. A prime is an exposure to a stimulus that influences a person's response to a later stimulus. It is a common technique used in psychology. Our current plan is to heighten the salience of polarization for a third of the sample, well- being for a third, and neither for the remaining third. Doing this, we will be able to tease out the "expectation effects" and apply that to the full study. (IE, if you think FB is bad for well being, how does being told the study is about well being impact the results, etc). For the full analytics plan, see [ HIGHLY CONFIDENTIAL (COMPETITOR) META3047MDL-019-00095649 Case 4:22-md-03047-YGR Document 2885-28 Filed 03/26/26 Page 5 of 7 People are [HYPERLINK "https://fb.workplace.com/not Self-reported biases es/ Compliance for deactivation Measurement Deactivation Confirmation Power /how-well- can-people-report-time-spent on-facebook-and-why-it- matters-to-well- bei/136722267538617/" \h ] how much time they spend -- most of the deactivation studies out there rely on self- reported data. We are aiming to have as high compliance rates as possible, but there are lots of opportunities for people to drop out of the study due to length, fatigue, challenges associated with deactivation, etc. Most of the deactivation studies that have been completed were completed via economics departments who do not specialize in areas like well-being. Without having access to log data (which we won't because we're trying to avoid having access to PII) it's really difficult to figure out if people have complied with deactivation. We're going after pretty small effect sizes here based on what we know so far. This means we have to be very careful about noise and sample size. HYPERLINK "https://docs.google.com/doc ument/d/17eMGfiV1ak5YugKl voqfr1DIP- kvnyK2CWyokP99jZs/edit" \h ]. Unbranded: We're also running the study unbranded through Nielsen, but will reveal that FB is the funder of the study at the end (and give people a chance to remove their data). Rely on mobile sensor data and FB/IG log data as much as we can. We will know how people substitute their time, if they use their phones differently, and we'll also have proxies for well-being (e.g., estimated sleep time, estimated travel time) Avoid complicated WTA, pay people per survey, AND deactivation. People are also only eligible for the study if they are willing to deactivate for $25 (an amount selected based on prior WTA studies that suggest FB's value is about $100/month). Reliance on top tier external instruments on each subject area. Worked with internal and external experts on these items (many of which we already use internally). We have written a script that determines first whether someone has an FBID, and then once we confirm an active account, we are looking for 404s of accounts. We've had to get special rate limiting privileges here to be able to do this without being blocked by our site. It is unclear how other larger studies have been able to do this at scale. Block randomization based on income (an attribute that's been correlated with topics of interest) Reliance on mobile sensor data where we'll have many points in time Going after more active FB folks Reducing noise by going after "high quality" survey folks who have to pass certain checks HIGHLY CONFIDENTIAL (COMPETITOR) META3047MDL-019-00095650 Case 4:22-md-03047-YGR Document 2885-28 Filed 03/26/26 Page 6 of 7 For a full list of our major decisions for this project, see [ HYPERLINK "https://docs.google.com/document/u/3/d/1IBU9TwBHoBamRBPI6fIIYyiou9lIK62jX5gmnllbVIg/e dit" \I "heading=h.bvnw2ynz7gzo" \h ]. Areas for improvement logistically moving forward (if possible): Get external advisory from various experts in each of the domains we're interested in Put through IRB/external review board Pre-register analyses (Note: [HYPERLINK "https://docs.google.com/document/d/17eMGfiV1ak5YugKIvoqfr1DIP- kvnyK2CWyokP99jZs/edit" \h ]) Publish findings externally Funding (We have made some decisions to help with funding (e.g., reducing sample size, only capturing metered data on a subset of people, etc.). Additional add ons for study: Micro-changes Larger sample size Mobile sensor data for everyone (currently just doing a subset for cost) Add iPhone users Macro-changes/additions: Deactivation of other apps and behaviors (e.g., YouTube, Snapchat, television) Expanding sensor data to computers, TVs, etc. Varying lengths of deactivation (1 week, 1 month, 3 months, 6 months, 1 year) Network effects (have multiple people in a network all deactivate together) Exploring expectation effects even further through different experiments on which works the best Exploring WTA in different ways Go beyond US to other countries Altering how people use FB (e.g., use it more actively, use it more passively, etc.) Altering deactivation fee Completely different studies (but similar goals): Augmentation study (e.g., in markets with lower penetration, have some people start using FB who never used it before) Big longitudinal survey panels ongoing (paired with log data) Ethnographic study following people around and observing how they use their phones, social media, etc. in various settings throughout their lives For more details: [HYPERLINK "https://docs.google.com/document/d/12xQO-I6yvL8VQs81CdQ- Z28bKsX8H8SxZ2z3Zk8w3UE/edit" \h ] HIGHLY CONFIDENTIAL (COMPETITOR) META3047MDL-019-00095651 Case 4:22-md-03047-YGR Document 2885-28 Filed 03/26/26 Page 7 of 7 Survey 1: [HYPERLINK "https://docs.google.com/document/d/1QdAV4iE8OMeQwnswDMWn578T0XIKKj4hjFsD 1M0IZ3U/edit" \h ] Survey 2: [HYPERLINK "https://docs.google.com/document/d/1Br8My2_72SXgmQ8Ekc13T7I7z- 14m2Jax81140m55NY/edit" \h] (scroll down to header "Survey Intros/Conclusions") [HYPERLINK "https://docs.google.com/spreadsheets/d/15_LY8jsXQIkYsclknX_g- gZVOu330h6d-sbfVRjdPzY/edit" \I "gid=1254379" \h] (focused on well-being, polarization, and social interaction proxies) [HYPERLINK "https://www.dropbox.com/s/36fxri4q0ell6zk/Facebook%20- %20Nielsen%20Custom%20Services%20SOW%20%28Well- Being%20Research%29%20%28Final%29%20-%20signed.pdf?dl=0" \h ] [HYPERLINK "https://docs.google.com/document/d/1IBU9TwBHoBamRBPI6fIIYyiou9IIK62jX5gmnlIbVI g/edit" \I "heading=h.bvnw2ynz7gzo" \h] Task: [HYPERLINK "https://our.internmc.facebook.com/intern/tasks/?t=29290761" \h] [HYPERLINK "https://drive.google.com/drive/u/3/folders/156OWAT6UFW3xfD6sPY8O2CH0BVDGmU WB" \h ] HIGHLY CONFIDENTIAL (COMPETITOR) META3047MDL-019-00095652