Bad Science Rebuttals Mixed-Methods Study

Raw text

Transcribed text as recorded, including headers and formatting fragments.

GRAVEYARD - TO PRESERVE Bad Science Rebuttals Mixed-Methods Study POC: Daniel Lauren Bryce Bartlett, Anett Bryan Emily Exhibit 51 MOP - 11/7/25 PLTF EX PLTF EX-04350_0001 PLTF EX-04350 Graveyard PLTF EX-04350_0002 Focus Group TL;DR - Insiders and parents in both the US and UK found the Haidt article credible because it reinforced what they already believe - that social media is bad for teen wellbeing. To parents (or others with teens in their lives) this is especially personal. They've seen young people suffer for the last few years, and believe social media is one important part of the problem. The Haidt article we showed them did not surprise them - but it did reinforce their existing beliefs. - As a result, none of the rebuttals we tested was effective at shifting their views on the impact of social media, because respondents felt they were being asked to disbelieve their own personal experiences. Specifically: a. The "heightened concern" message compared the panic over social media to past panics about the radio or comic books. This rebuttal triggered resentment because it suggested parents' concerns were irrational. There was skepticism towards comparing social media to the radio or comic books (for example, due to concerns about bullying on social media). Additionally, no one has seen enough action from lawmakers or social media companies to count as a "panic." o. The "bad science" rebuttal highlighted issues with Haidt's statistical analysis. This message missed the point for most respondents: it nitpicked methodology without addressing Haidt's conclusions (or parents' concerns). Left-leaning Insiders in both the UK/US found the data-based arguments compelling, but it did not change their minds on the problem. Other audiences resented the rebuttal for focusing so narrowly on research methods. - The "tools and autonomy" rebuttal said society should support teens so they can learn how to interact online, rather than coddling them. The suggestion that "we should not coddle teens" triggered backlash in every group. While most agreed it is society's collective responsibility to provide teens tools and skills to prepare them, this message did not strongly enough acknowledge the responsibility of social media companies and what they are doing to help. To be effective, this rebuttal would need to be optimized to place even more responsibility on social media companies. PLTF EX-04350_0003 TL;DR (cont.) - There are individual elements of the various rebuttals which, when combined, might result in a more effective argument. This includes: o Suggesting more research needs to be done. o Pointing out that social media provides some important benefits to teens. o An acknowledgement that there nevertheless is a problem, and society has a collective responsibility to support teens. o Affirming that social media companies must play a central role in this effort and highlighting what they are doing to address concerns. - However, panelists did not think these rebuttals would be credible coming from Meta. The rebuttals would likely be perceived as even less convincing and in particular the "tools and autonomy" and "heightened concern" messages would be perceived as inappropriate or dismissive. Respondents believe they've been misled by Meta about teen wellbeing, and point to the whistleblower story that the company ignored its own research on the subject. - Though explored only in the UK, we found that parents and Insiders are willing to listen to independent research based on Meta's data. But there are concerns about Meta cherry-picking the data, so an independent researcher would need to vouch for the data and Meta's transparency during the process. Even if the data was analyzed by a third-party, independent researcher, respondents would wonder if Meta had provided them with all of the relevant data. They would want assurances from the researcher that this was the case before they take the conclusions seriously. 4 PLTF EX-04350_0004 Research Objectives This research investigated the effectiveness of rebuttals to an argument from Jonathan Haidt in The Atlantic that there is strong evidence pointing to the fact that social media is the most important factor contributing to the deterioration of teen mental health. 1. After sharing Haidt's article presenting this argument, we discussed reactions to 3 rebuttals displayed as coming from psychologists: a. Rebuttal 1: Heightened concern termed 'panic' in scientific literature b. Rebuttal 2: Teens shouldn't be sheltered but given tools and autonomy c. Rebuttal 3: Bad science 2. We were most interested in learning whether the rebuttals: a. Resonate and are persuasive against the arguments presented in The Atlantic article b. Are seen as appropriate for Meta to share c. Receive backlash, including increased punitive and/or negative sentiment toward Meta 3. Finally, we were interested in gauging the credibility of different research sources that could be used in rebuttals, specifically: a. A social media company citing its own internal research and data b. A social media company citing independent, third-party research and data c. An independent, third-party researcher using data that was collected and provided to them by a social media company 5 PLTF EX-04350_0005 All audiences believed social media is a leading factor harming teen wellbeing, and they say they see it with their own eyes. While respondents acknowledged that lockdowns, bullying, and other issues have contributed to the mental health problems facing teens, social media is high on the list. For parents, this is especially personal. "I mean, I related to it, because my daughter had anxiety and depression. And I think a lot had to do with not feeling good enough or not feeling worthy enough, or pretty enough or skinny enough? And, and, you know, looking at all [unrealistic content on social media], I think that it definitely does something to a child's psyche." - US HINE Parent "I think on social media, the amount of photographs that are uploaded, and the usage of these apps that make people's skin look perfect, and they can throw in the waist or define features that's affecting girls mental health, for sure. Because that's my kids that I know, and my daughter being very insecure ... She's always looking at things on there. And because of that, she's trying to keep up." - UK HINE Parent "We're really concerned about social media, and in particular, our daughter with regard to body shape, body form. Occasional hyper-sexualization of a woman, and also more generally, they can have the idealized picture that appeals to people's lives. Instagram is particularly poisonous." - UK Left-leaning Insider "For my children in their mid-twenties ... their awareness of the damage [social media] has caused them and is causing, and could potentially cause them is increased as they've gotten older. So they're starting to use strategies to help cope with that ... and engage in ways they didn't earlier on" - UK Right-Leaning Insider 6 PLTF EX-04350_0006 Because respondents already believed social media harms teens, Haidt's article didn't surprise them. Some respondents had questions about a few of the stats referenced in the article. But generally, respondents didn't need verified data to agree with the article's underlying premise. - Parents, especially those of daughters, strongly agreed with the claim that social media has led to greater depression/anxiety. Many cited the elbow curve statistic as very convincing. - UK respondents were more skeptical of the research referenced by Haidt - whether sample size, time period, or saying something was true in the absence of an alternative explanation. But they still largely agreed with his argument overall. - A few respondents noted that social media may be one factor negatively affecting or exacerbating teen wellbeing, but that saying it was the main or sole cause felt like an overstatement. - Respondents were also curious about what next steps Haidt recommended. A few assumed it made a case for additional government oversight - which they felt was necessary and long overdue. "[I react to this article with] not a lot of shock. It's pretty obvious that this generation is a lot of stress and anxiety and depression that they're feeling and that social media certainly plays a role in that." - US Democratic Insider "It was very depressing to read the bit where it said that it's questionable how much proof regulators need before they'll intervene, which is a shame because I'm thinking, 'Well, how much do they need?' When's it going to stop because obviously, it's very severe, already." - UK HINE Parent "The data is is really out of date ... just horrendously out of date, because we should have much more current information ... things have moved on since then." - UK Right-Leaning Insider "So I wanted to get to the conclusion of what the author, you know, was trying to get to because it feels like they're going to have some kind of recommendation, but I'm skeptical of what that might be." - US Republican Insider 7 PLTF EX-04350_0007 Rebuttal Deep-Dive PLTF EX-04350_0008 Parts of the "Tools and autonomy" message were effective, but the overall message backfired due to language about "coddling teens." Parts of this rebuttal resonated with respondents because it acknowledged there was a problem and emphasized collective responsibility to support teens. What Worked: Collective responsibility. Respondents agreed that this problem won't be solved by one party alone and that parents, social media companies, and policymakers all have a role to play. - "I also agree that teens need to be prepared to deal with social media and technology. I also agree that it is our collective responsibility to provide teenagers with tools and skills to help them navigate what they may encounter online and offline." - US HINE Parent Providing tools and skills. Respondents thought preparing teens to navigate the online world was a way social media companies and parents can protect teens while preserving the benefits of social media. - "And now I think social media companies definitely have a responsibility to provide kind of information and tools, you know, they are the creators of this virtual world and therefore, we would rely on them for a level of virtual policing." - UK Right-Leaning Insider What Didn't Work: Passing the buck. Some respondents thought the rebuttal gave too much of a pass to social media companies. They felt social media companies could currently do more to protect teens, but choose not to. - "I did think that it was a bit of a swerve though. And I think what this misses is that technology companies should be doing a lot more to protect teenagers, so not just in terms of controlling fake news, but also dealing with online abuse ... the technology companies have actually got the tools to do more, but they're not." - UK HINE Parent "Coddling teens" language. Audiences overwhelmingly reacted negatively to the word "coddle," viewing it as dismissing the problem. Parents were particularly upset, and said it's not coddling to care about their teen's health. - "[The use of 'coddling'] was the first thing I had an extremely strong reaction to - teens or children, like, their brains are literally not developed yet. So that's just parenting - not coddling. And that really bothered me." - US Democratic Insider Lacks substance. Respondents described this rebuttal as shallow and lacking in substance. They felt it made a broad, ethical point on "collective responsibility" without seriously responding to the specific concerns about social media put forth by Haidt. - "I find that piece of text unpersuasive, bland, relatively directionless. It's society's collective responsibility to teach people ... However young one can start and encourage them to think ... Well, good luck with that." - UK Left-Leaning Insider PLTF EX-04350_0009 The "heightened concern" rebuttal felt dismissive, did not match panelists' lived experiences, and triggered resentment. Parents especially resented the implication they were irrationally panicking. And respondents did not see other parties - companies, legislators, etc. - taking any action, let alone "panicking." What Worked: Mentioning the benefits of social media. Respondents acknowledged the educational and communication benefits of social media, especially during the pandemic. - "I liked the point [about] appreciating that children have the privilege of technology to stay connected, educated and informed. I think, as I said before, that they suddenly are part of a global community, whatever their particular interests or friendship groups or whatever, those now exist, and they can be part of them. And I think that's got to be good for their confidence and their happiness." - US Democratic Insider More concrete research is needed and parents deserve science. Most respondents expressed a desire to see more serious research on this issue. - "'They deserve the science' - I think that's persuasive. I think we're all kind of fed up with that kind of moral panic and would rather have more facts and statistics than just people's opinions." - UK HINE Non-Parent What Didn't Work: Implication of irrational panic. Most thought the "moral panic" angle undermined their lived experiences with social media. Parents especially didn't think there was enough of a panic - from lawmakers or tech companies - surrounding the severe effects of social media they witnessed in their children. - "The language used is things like 'the panic around social media' - which is kind of insulting really to those who have been just careful and quite rightly concerned. And it's to say that the effects are small and practically meaningless." - UK HINE Parent Comparing social media to other forms of media. Although it resonated for a small group, most respondents didn't buy the comparison of social media with comic books/video games, stressing that social media's all-consuming interactivity sets it apart. - "The radio is not social media, comic books are not social media, the way that you use those things is in no way comparable to how people are using social media." - US HINE Parent Lacks substance. Respondents felt this rebuttal made overly-defensive claims and relied on weak examples. Though its evidence was up for debate, Haidt's article came off as better informed. - "[In Dr. Haidt's article] there was an abundance of fact compared to Dr. Kowert who was positively breathless in attacking that without much in the way of actual data other than to say 'you don't have any' ... So there was nothing really here of any kind of substance, other than assertions." - UK Left-Leaning Insider "This isn't about screen time, this is about me engaging on a network, as opposed me playing a computer game, or listening to music, etc... The difference is, always, that with social media, it's about me, because I'm the product." - UK Right-Leaning Insider "I didn't see a lot of actual hard evidence behind these rebuttals. I mean, at least the [Haidt] article, which I had my skepticism to that one to, at least that points to an actual research study ... like that had some concrete arguments behind it. This was really kind of more general, and that makes me more skeptical." - US Republican Insider 11 PLTF EX-04350_0010 By focusing more on research methods than parents' concerns, the "bad science" rebuttal felt dismissive. Most respondents felt it was a transparent attempt to undermine the data Haidt referenced, rather than seriously grappling with the issues he raised. What Worked: Calling for more evidence. Like Dr. Orben, respondents wanted to see specific and credible studies that support the causal links between social media use and poor teen mental health. - "Because I agree that the the evidence is weak, I need to see more data driven studies that have been validated to show cause and effect ... In my opinion, from what I hear is, even my own personal experiences, I do think that decreases self esteem, that there's some issues, but I don't think there's been any really good concrete studies that verify that." - US HINE Non-Parent Making a data-driven argument. In particular, left-leaning Insiders in both the US and UK thought the methodical framing of this response made it the most persuasive of the three. - "[This article is] pointing to specific studies and data to make the argument and it also directly attacks the studies used in the other article. So for that reason, I found it far more persuasive." - US Republican Insider What Didn't Work: Avoiding the issue. While some acknowledged more research needed to be done, many felt that this rebuttal relied too heavily on data nitpicking, and failed to confront the broader issue of teen wellbeing. - "The psychologists are starting to argue about who has the better studies, ... I feel like they're kind of sniping at each other and just kind of making jabs and you don't really get too serious points ... I just find them dismissive on a surface level." - US Republican Insider Pushing for more rigorous evidence while avoiding citations. Many respondents were puzzled that a counterargument attempting to discredit another researcher's data failed to properly cite its own evidence. - "There's no statistics contained in any of this, there's nothing that we can refer to, we're just being given a load of blardy blar, but without any facts to back them up." - UK HINE Non-Parent - "I would want to see the research data [on these counter studies] so that I could make my own evaluation ... everything we've read inevitably is an interpretation by the author." - UK Left-Leaning Insider PLTF EX-04350_0011 All audiences were open to some elements of the rebuttals, specifically: 1. Suggesting more research needs to be done. Respondents agreed more data is needed to definitely answer and better understand the relationship between social media and teen well-being. - "I do think a stronger case can be made to link social media and teen wellbeing. I think that that's going to take literally forcing social media companies to give us data or give researchers data to do that. So I think it's fair to say that we need more information to really draw like super strong conclusions." - US HINE Parent 2. Pointing out that social media provides some important benefits to teens. Respondents appreciated the opportunities enabled by social media. Generally, it wasn't a question of whether teens should access social media, but whether there are the guardrails in place to safely do so in moderation and appropriate ways. - "[The benefits of social media for teens] include connecting to friends, learning opportunities, different ways that it's actually improving people's lives, especially people in rural areas that might not have as much access to being physically around other people. There's a lot to talk about." - US Republican Insider 3. An acknowledgement that society has a collective responsibility to support teens. Respondents admitted that no single entity could effectively look over teens' shoulders all the time, and as such, diffused responsibility across lawmakers, parents, and social media companies. - "I think that it is society's collective responsibility to support teens. It takes everybody, doesn't it? The parents, government, the companies, everyone working together to make things safe." - UK HINE Parent 4. Affirming that social media companies must play a central role in this effort. While respondents acknowledged that everyone has a role to play, they believe social media companies are at the top of the list. They want companies to take action to make their platforms safer and empower parents to protect their children. - "I think [social media companies] have a high level of responsibility. I think they earn so much money, and they are so ubiquitous in people's lives, that some of that, a significant portion of those resources do need to go towards mitigating negative effects, either, you know, people purposely doing it or inadvertently ... The companies need to play a huge part." - UK Right-Leaning Insider 12 PLTF EX-04350_0012 The rebuttals were even less effective when respondents were asked to imagine them coming from Meta or other social media companies: - Panelists do not trust Meta on this topic, so they question the validity of any message from Meta. The whistleblower stories have stuck in their memory and have led them to believe Meta won't be transparent about this issue. - Panelists believe Meta is too self-interested. Respondents believe Meta's business model depends on keeping teens on social media. As a result, they wouldn't believe Meta's assertions about teen wellbeing. - The "heightened concern" and "tools and autonomy" messages in particular would not be an appropriate tone coming from Meta or other social media companies. "Just because of what's been happening the last 10-15 years ... [Meta] keep[s] on coming forward with 'yeah, we're gonna clean up our act,' and there's more whistleblowing, then you find out they haven't done anything - it's actually worse than we thought. So anything coming from Meta, I would disregard." - UK HINE Non-Parent "I just don't think [Meta making one of these arguments] would be that credible ... I just think they profit pretty much the most owning Instagram. So it's in their best interest for people not to want to do anything to limit the access they have to people of all ages." - US Republican Insider "I think both ['heightened concern' and 'tools and autonomy'] coming from a social media company would immediately engender people to just completely reject it ... raising questions around this 'coddling teens' and this idea that it's 'panic' is, I think both of those ideas are anathema to me, and they would be particularly anathema coming from social media [companies]." - US Democratic Insider 13 PLTF EX-04350_0013 Appendix PLTF EX-04350_0014 Research/Data Sources Credibility PLTF EX-04350_0015 UK respondents were willing to listen to independent, third-party research based on Meta's internal data, but would have questions about its credibility. A social media company citing independent, third-party research and data: Respondents thought this combination method would be the most credible of the three. Nevertheless, many respondents said they'd assume the third party was paid to conduct the research by the social media company. - "I think in this day and age for, you know, the modern day discussion on the court of public opinion, I think you need independent and other sort of third-party validators." - US Democratic Insider An independent, third party researcher using data that was collected and provided to them by a social media company: Respondents said they'd be open to this kind of research, and that they'd find it credible to the extent that the researcher was truly independent. A few stressed that while social media companies should publicly share their internal data, they'd be wary of it being cherry-picked. - "As long as the data given was robust and credible and hadn't been manipulated in any way, then I think [I would] absolutely [trust data from an] independent third party researcher as well ... if there were appropriate safeguards in place, then I would find that very strong." - UK Right-Leaning Insider A social media company citing its own internal research and data: This research ranked last on credibility. Respondents likened it to "marking your own homework." As such, they'd reject this kind of research findings out of hand. - "[A social media] company citing its own internal research and data is [like] marking your own homework is a great way of putting it. You know, you'll be very selective about the data you use, and you'll be selective to try and make sure to try and make the situation a lot more positive in favor of social media companies." - UK HINE Parent 16 PLTF EX-04350_0016 In order to trust an independent, third-party study based on Meta's data, respondents emphasized that the researcher would need to vouch for the data and the company's transparency during the process. UK HINE Parents: For UK HINE Parents, lack of trust was the key issue with this research method. They worried about a SM company cherry-picking data provided to advance its own narrative. The group did prefer this method to a SM company citing its own internal research, but would need full disclosure on how the data was collected and shared. - "My only issue with that is that they could obviously pick and choose what data they wanted to provide to the researcher. But it sounded better than the other [research method: SM company citing its own research] to me." - UK HINE Parent - "[My concern is] data manipulation - they could just give them the data, that looks good and fits good [with the company narrative]." - UK HINE Parent - "You can cherry-pick what you want to show the third-party company. And I think what's the fundamental issue at the heart of all this, is trust." - UK HINE Parent UK HINE Non-Parents: UK HINE Non-Parents expressed concerns about transparency and research independence. They put the onus on the third-party researcher to ensure the integrity of the data. So long as the SM company provided "unfettered access" to the data, this group generally found this method credible. - "So, yeah, they are going to be selective about the data. But I think that's the role of the third-party researcher to make sure that they're challenging them on that. That's why even though that's a concern, it's not a big concern." - UK HINE Non-Parent - "I would have to ask [the third-party researcher] if they had basically unfettered access to the data rather than selected." - UK HINE Non-Parent - "[I would like to know] if the researcher had the [ability] to basically sift through the data and see if they are being manipulated or not." - UK HINE Non-Parent UK Right-Leaning: Similarly, UK Right-Leaning Insiders would find this research method credible if the third-party researcher could corroborate that the data the SM company shared was complete and not bent toward company interests. They welcomed the idea of SM companies opening up their data and allowing independent parties to interpret it. - "I think the key thing would be whether the researcher had the ability to specify the data, and whether they were they were confident that it was complete." - UK Right-Leaning Insider - "For me [it is important to know] whether the data was credible and hadn't been manipulated by the social media company to achieve its aims." - UK Right-Leaning Insider - "Collecting data from a company and then analyzing it - using that as content is slightly different than a social company just coming up with his own research on interpreting. So there's an interpretation of data ... I would encourage [SM companies] to perhaps, maybe open up some of their data for the people to interpret." - UK Right-Leaning Insider UK Left-Leaning: UK Left-Leaning Insiders were most skeptical about the credibility of data provided by SM companies and the third party behind this kind of research. At least one respondent completely dismissed the use of internal company data, while others would want to know if the third party had control over the research design and type of data shared. - "Any academic researcher worth their salt would want to be responsible for the design, distribution and analysis of the data and to work on something that's been provided by a social media company or any third party, I think they just wouldn't accept it. It just wouldn't be credible in academic circles." - UK Left-Leaning Insider - "I would want to know a bit more about the actual independence of the researcher, and where their funding was coming from, what data they'd accessed, whether it was just presented to them by the platform, or whether they got everything they asked for, and also how they knew what to ask for." - UK Left-Leaning Insider - "That data will be bent, that data will be selected, that data will be partial." - UK Left-Leaning Insider PLTF EX-04350_0017 Appendix PLTF EX-04350_0018 Appendix: Survey Primes and Messages Tested PLTF EX-04350_0019 Appendix: Stim Tested in Focus Groups PLTF EX-04350_0020 Criticism Against Social Media Companies (Dr. Haidt) Psychologist Dr. Haidt writes in the Atlantic: Social media—particularly Instagram, which displaces other forms of interaction among teens, puts the size of their friend group on public display, and subjects their physical appearance to the hard metrics of likes and comment counts—takes the worst parts of middle school and glossy women's magazines and intensifies them. One major question, though, is how much proof parents, regulators, and legislators need before intervening to protect vulnerable young people. The preponderance of the evidence now available is disturbing enough to warrant action. The available evidence suggests that Facebook's products have probably harmed millions of girls. If public officials wanted to make that case, it could go like this: 1. Harm to teens is occurring on a massive scale. Something terrible has happened to Gen Z, the generation born after 1996. Rates of teen depression and anxiety have gone up and down over time, but it is rare to find an "elbow" in these data sets—a sharp increase occurring in a short time span of just two or three years. Yet when we look at what happened to American teens in the early 2010s, we see many such "elbow" turning points, usually sharper for girls. 2. The timing points to social media. By 2014, 80 percent of high-school students said they used a social-media platform on a daily basis. Of course, teens had long been texting each other, but from 2010 to 2014, high-school students moved much more of their lives onto social-media platforms. Notably, girls became much heavier users of the new visually oriented platforms, primarily Instagram (which by 2013 had more than 100 million users), followed by Snapchat, Pinterest, and Tumblr. 21 PLTF EX-04350_0021 3. The victims point to Instagram. The evidence is not just circumstantial; we also have eyewitness testimony. In 2017, British researchers asked 1,500 teens to rate how each of the major social-media platforms affected them on certain well-being measures, including anxiety, loneliness, body image, and sleep. Instagram scored as the most harmful, followed by Snapchat and then Facebook. Facebook's own research, leaked by the whistleblower Frances Haugen, has a similar finding: "Teens blame Instagram for increases in the rate of anxiety and depression." A recent experiment confirmed these observations: Young women were randomly assigned to use Instagram, use Facebook, or play a simple video game for seven minutes. The researchers found that "those who used Instagram, but not Facebook, showed decreased body satisfaction, decreased positive affect, and increased negative affect." 4. No other suspect is equally plausible. Many things changed in the early 2010s. Some have suggested that the cause of worsening mental health could be the economic insecurity that followed the 2008 global financial crisis. But why this would hit younger teen girls the hardest is unclear. Besides, the American economy improved steadily in the years after 2011, while teen mental health deteriorated steadily. Some have suggested that the 9/11 attacks, school shootings, or other news events turned young Americans into "generation disaster." But why, then, do similar trends exist among girls in Canada and the U.K.? Correlation does not prove causation, but nobody has yet found an alternative explanation for the massive, sudden, gendered, multinational deterioration of teen mental health during the period in question. 22 PLTF EX-04350_0022 First Rebuttal: Heightened Concern from Third Party (Dr. Kowert) Psychologist Dr. Kowert in response to the Atlantic article: a. As expected of mainstream journalism it is far more moral panic than actual information. It's a lot to digest and for the sake of brevity, I'm going to just hit the major points here. b. I think the part that got edited out was how much parents appreciate that their children have the privilege of technology to stay connected, educated, and informed. Unlike those who do not and who are being disproportionately left behind. c. While the Atlantic article is a great way to instill fear in us all, there is actually a LOT of debate in the scholarly community about the long-term effects of technology use on long-term, structural brain change. d. Therefore, it's really hard to make a case that the linkage between social media and teen well-being is so negative and strong. e. If you look to history, this is the latest chapter in a long line of misguided pundits peddling panic, like a claim that "listening to the radio causes anxiety" in the 1940s, or another claim that "reading comic books causes childhood maladjustment" in the 1950s. More recently, there was a panic about violent video games like Grand Theft Auto. Just like then this panic is not supported by strong evidence or justified. f. I'm not saying that there are NO consequences to this increase in screen time, but this is the most unbalanced, moral panic, fear-mongering article I've read in awhile. Parents deserve a more accurate and balanced overview of what we actually know. They deserve the science. We are tired of the moral panic about technology. 23 PLTF EX-04350_0023 Second Rebuttal: Tools and Autonomy (Dr. Howard) Psychologist Dr. Howard in response to the Atlantic article: a. We should not coddle teens-social media and technology is a core part of life nowadays and what they will encounter as adults. Teens need to be prepared for that and not left behind. b. Instead, it's society's collective responsibility to support teens and provide them with tools and skills to help navigate what they may encounter online and offline. Online this includes how long to be online, what to look at, and how to deal with difficult people. Social media companies must provide tools to support parents and teens to help them gain these skills while still empowering teens to leverage social media to connect with others, partake in education, and express themselves as healthy grown-ups. *Note: The order of points (a) and (b) were switched for the UK groups. 24 PLTF EX-04350_0024 Third Rebuttal: Scientific Rebuttal (Dr. Orben) Psychologist Dr. Orben in response to the Atlantic article: a. Evidence that social media makes people more depressed is very weak. The narrative that screens are killing our kids and teens, however, is strong and persuasive. I'm going to unpack some of the work cited in this Atlantic article. b. The studies used to support this narrative in the Atlantic article show trivially small effect sizes, suggesting little practical impact. They also rely on correlations and are not causal, and only apply to a small subset of people in the studies. c. Nothing about these studies lets us conclude that scrolling Facebook or Instagram makes you depressed. The majority of people in these studies were using social media at levels that did not link to depression. d. There's also a lot of disconfirming evidence missing from this Atlantic article, like a recent paper showing that the strength of the social media-mental health association has become weaker year over year. And another paper emphasizing the trivially small impact of technology use on adolescent well-being. And another one showing no daily associations between digital technology use (including social media) and depression, among other things. e. I could go on. Take-home points are: (1) it is possible to statistically show, generally in big survey studies, that extreme social media use is linked to poorer mental health, but (2) effects are so small as to be practically meaningless, and (3) often don't replicate. f. It's probably not a good idea to spend most of your waking hours on social media and extreme behaviors of any kind are concerning. Most of the panic around social media use is unwarranted and doesn't reflect the level of use typical of most young people. 25 PLTF EX-04350_0025 Appendix: Quantitative Study results PLTF EX-04350_0026 Impact on Teen Well-being - 1 When it comes to factors negatively affecting teen well-being, do you think social media is [the most important factor]? Proportion said social media is the "Most important factor" — chart comparing Tools and Autonomy, Heightened Concern, Scientific Rebuttal, Control, and Overall groups (All/UK/US), baseline vs. post-exposure. 27 PLTF EX-04350_0027 Impact on Teen Well-being - 2 Proportion of Adults Asserting SM is the most important factor for teen WB (scatter/dot plot with 95% CI, Baseline Value: One of many/Not a factor vs. The most important factor; Experimental Group: Control, Scientific Rebuttal, Heightened Concern, Tools and Autonomy; x-axis: Proportion selecting Most important Factor, split by Parent of Teen: Yes/No). 95% CI shown. From weighted linear probability model on full data. PLTF EX-04350_0028 Survey results PLTF EX-04350_0029 Limit Meta's ability to operate To what extent do you support or oppose taking government action to severely limit Meta's ability to operate in your country? Proportion selecting "Strongly or Somewhat Support" — chart comparing Tools and Autonomy, Heightened Concern, Scientific Rebuttal, Control, and Overall groups (All/UK/US), baseline vs. post-exposure. PLTF EX-04350_0030 Limit Meta's ability to operate Proportion of Adults Asserting supporting taking government action to severely limit Meta's ability to operate (scatter/dot plot with 95% CI; Baseline Value: Somewhat or Strongly oppose or Unsure vs. Strongly or Somewhat Support; Experimental Group: Control, Scientific Rebuttal, Heightened Concern, Tools and Autonomy; x-axis: Proportion selecting Strongly or Somewhat Support, split by Parent of Teen: Yes/No). 95% CI shown. From weighted quasibinomial probability model on full data. Question: To what extent do you support or oppose taking government action to severely limit Meta's ability to operate in your country? PLTF EX-04350_0031 Objectives and Methodology YouGov's Methodology Sampling Methodology - Off-platform survey; web-based interviewing using YouGov's RealTime platform - N = 2,123 (UK); 2,544 (US) - Field dates: August 15-16, 2022 Analytical Methodology - Significance testing (Experimental outcomes 1 and 2 in the Topline): o Comparing arithmetic change (measured as means) from baseline to post exposure across experimental groups for primary and secondary outcomes of interest o Comparing percentage differences (measured as means) for post-exposure only across experimental groups for primary and secondary outcomes of interest PLTF EX-04350_0032 Pre-Post Significance Testing PLTF EX-04350_0033 Appendix: Yougov Slides PLTF EX-04350_0034 Pre-Post Numeric Differences (US) Respondents exposed to the CSN Control Message perceive Meta to be more committed to teen well-being than respondents who saw any other message. [Data table - Outcome / DV / Mean / Group A-CSN Control / Group B-Scientific Rebuttal / Group C-Heightened Concern / Group D-Tools and Autonomy] Wellbeing Impact Problem Difference: 1.1.1 Mean .05 (A .06, B .05, C .06, D .04); 1.1.2 Mean .04 (A .03, B .03, C .03, D .07) Wellbeing Problem Ranking Difference: 1.2.2 Mean .23 (A .22, B .17, C .28, D .27); 1.2.3 Mean .08 (A .08, B .05, C .09, D [blank]) Limit Difference: 2.1.1 Mean .06 (A .05, B -.02, C .07, D .15[sig B]); 2.1.2 Mean .06 (A .04, B .04, C .06, D .09[sig B]) Good for World Difference: 2.2.1 Mean -.07 (A -.03[sig D], B -.02[sig D], C -.05[sig D], D -.17); 2.2.2 Mean -.01 (A .01, B .02[sig D], C -.02, D -.03) Committed Difference: 2.3.1 Mean .02 (A .22[sig B,C,D], B -.03, C -.06, D -.06); 2.3.2 Mean .04 (A .16[sig B,C,D], B .01, C 0, D .01) N: Mean=2,123; Group A=541; Group B=519; Group C=518; Group D=545. Superscript letters (A,B,C,D) indicate significance at 95% confidence in pairwise testing. PLTF EX-04350_0035 Pre-Post Numeric Differences (UK) Mirroring the US findings, respondents in the CSN Control group have significantly higher means on perceptions of Facebook's commitment to teen well-being compared to all other messages. [Data table - Outcome / DV / Mean / Group A-CSN Control / Group B-Scientific Rebuttal / Group C-Heightened Concern / Group D-Tools and Autonomy] Wellbeing Impact Problem Difference: 1.1.1 Mean .05 (A .06, B .03, C .05, D .07); 1.1.2 Mean .05 (A .05, B .02, C .04, D .06) Wellbeing Problem Ranking Difference: 1.2.2 Mean .19 (A .29[sig B,C], B .11, C .14, D .20); 1.2.3 Mean .08 (A .11[sig B], B .06, C .08, D .08) Limit Difference: 2.1.1 Mean .06 (A .09[sig C], B .04, C -.02, D .11[sig C]); 2.1.2 Mean .05 (A -.07, B -.05, C -.02, D .07[sig C]) Good for World Difference: 2.2.1 Mean -.10 (A -.10, B -.17, C -.03[sig B,D], D -.12); 2.2.2 Mean 0 (A .01, B 0, C .02, D -.01) Committed Difference: 2.3.1 Mean .03 (A .19[sig B,C,D], B -.10, C .03[sig B], D .01[sig B]); 2.3.2 Mean .05 (A .13[sig B,C,D], B 0, C .04[sig B], D .02) N: Mean=1,882; Group A=485; Group B=463; Group C=483; Group D=471. Superscript letters (A,B,C,D) indicate significance at 95% confidence in pairwise testing. PLTF EX-04350_0036 Post Only Significance Testing PLTF EX-04350_0037 Post Only Numeric Differences (US) Those who saw Tools and Autonomy are more likely than those who saw Scientific Rebuttal or Heightened Concern to rank well-being as the most important negative effect of social media. [Data table - Outcome / DV / Level / All / Group A-CSN Control / Group B-Scientific Rebuttal / Group C-Heightened Concern / Group D-Tools and Autonomy] Wellbeing Impact Problem Difference (1.1.2): The most important factor - All 29%, A 26%, B 27%, C 29%, D 32%[sig A]; One of many/Not a factor - All 71%, A 74%, B 73%, C 71%, D 68% Wellbeing Problem Ranking (1.2.2): Rank 1 - All 41%, A 41%, B 39%, C 40%, D 45%[sig B,C]; Rank 2 - All 19%, A 20%, B 19%, C 20%, D 18%; Rank 3 - All 15%, A 15%[sig D], B 16%[sig D], C 15%[sig D], D 11%; Rank 4 - All 10%, A 8%, B 12%[sig A,C], C 8%, D 10%; Rank 5 - All 7%, A 7%, B 7%, C 9%[sig D], D 6%; Rank 6 - All 8%, A 7%, B 7%, C 8%, D 9% N: Mean=2,544; Group A=622; Group B=630; Group C=648; Group D=644. Letters (A,B,C,D) indicate significance at 95% confidence in pairwise testing. PLTF EX-04350_0038 Post Only Numeric Differences (US) Respondents who saw messages other than CSN Control were more likely to say that Meta is not committed or that they are unsure of Meta's commitment to teen well-being. [Data table - Outcome / DV / Level / All / Group A-CSN Control / Group B-Scientific Rebuttal / Group C-Heightened Concern / Group D-Tools and Autonomy] Limit Post (2.1.2): Support - All 49%, A 47%, B 49%, C 50%, D 51%; Unsure/Oppose - All 51%, A 53%, B 51%, C 50%, D 51% Good for the World Post (1.1.2): Agree - All 18%, A 21%, B 17%, C 19%, D 17%; Neutral/Disagree - All 82%, A 79%, B 83%, C 81%, D 83% Committed Post (2.3.2): Committed - All 30%, A 42%, B 24%, C 26%, D 27%; Unsure/Not Committed - All 70%, A 58%, B 76%[sig A], C 74%[sig A], D 73%[sig A] N: Mean=2,544; Group A=622; Group B=630; Group C=648; Group D=644. PLTF EX-04350_0039 Post Only Numeric Differences (UK) Respondents exposed to the CSN Control message are more likely than those exposed to Heightened Concern to say well-being is the most important negative effect of social media. [Data table - Outcome / DV / Level / All / Group A-CSN Control / Group B-Scientific Rebuttal / Group C-Heightened Concern / Group D-Tools and Autonomy] Wellbeing Impact Problem Difference (1.1.2): The most important factor - All 32%, A 32%, B 32%, C 32%, D 34%; One of many/Not a factor - All 68%, A 68%, B 68%, C 68%, D 66% Wellbeing Problem Ranking (1.2.2): Rank 1 - All 53%, A 56%[sig C], B 55%, C 49%, D 53%; Rank 2 - All 17%, A 18%, B 14%, C 19%[sig B], D 19%[sig B]; Rank 3 - All 13%, A 12%, B 13%, C 13%, D 12%; Rank 4 - All 7%, A 6%, B 8%, C 8%, D 6%; Rank 5 - All 5%, A 4%, B 5%, C 7%, D 5%; Rank 6 - All 4%, A 3%, B 5%, C 4%, D 3% N: Mean=2,123; Group A=541; Group B=519; Group C=518; Group D=545. PLTF EX-04350_0040 Post Only Numeric Differences (UK) Similar to the US, respondents who saw messages other than CSN Control are more likely to say that they are unsure or that Meta is not committed to teen well-being. [Data table - Outcome / DV / Level / All / Group A-CSN Control / Group B-Scientific Rebuttal / Group C-Heightened Concern / Group D-Tools and Autonomy] Limit Post (2.1.2): Support - All 56%, A 58%, B 54%, C 55%, D 59%; Unsure/Oppose - All 44%, A 42%, B 46%, C 45%, D 41% Good for the World Post (1.1.2): Agree - All 12%, A 13%, B 11%, C 13%, D 14%; Neutral/Disagree - All 88%, A 87%, B 89%, C 87%, D 86% Committed Post (2.3.2): Committed - All 22%, A 30%[sig B,C,D], B 18%, C 20%, D 18%; Unsure/Not Committed - All 78%, A 70%, B 82%[sig A], C 80%[sig A], D 82%[sig A] N: Mean=2,123; Group A=541; Group B=519; Group C=518; Group D=545. PLTF EX-04350_0041 TL;DR - Insiders and parents in both the US and UK found the Haidt article credible because it reinforced what they already believe - that social media is bad for teen wellbeing. To parents (or others with teens in their lives) this is especially personal. They've seen young people suffer for the last few years, and believe social media is one important part of the problem. The Haidt article we showed them did not surprise them - but it did reinforce their existing beliefs. - As a result, none of the rebuttals we tested was effective at shifting their views on the impact of social media, because respondents felt they were being asked to disbelieve their own personal experiences. Specifically: o The "heightened concern" rebuttal triggered resentment because it suggested parents' concerns were irrational. There was skepticism towards comparing social media to the radio or comic books (for example, due to concerns about bullying on social media). Additionally, no one has seen enough action from lawmakers or social media companies to count as a "panic." o The "bad science" rebuttal missed the point for most respondents: it nitpicked methodology without addressing Haidt's conclusions (or parents' concerns). Left-leaning Insiders in both the UK/US found the data-based arguments compelling, but it did not change their minds on the problem. Other audiences resented the rebuttal for focusing so narrowly on research methods. - In the "tools and autonomy" rebuttal the suggestion that "we should not coddle teens" triggered backlash in every group. While most agreed it is society's collective responsibility to provide teens tools and skills to prepare them, this message did not strongly enough acknowledge the responsibility of social media companies and what they are doing to help. To be effective, this rebuttal would need to be optimized to place even more responsibility on social media companies. 42 PLTF EX-04350_0042 TL;DR (cont.) - There are individual elements of the various rebuttals which, when combined, could result in a more effective argument. This includes: o Suggesting more research needs to be done. o Pointing out that social media provides some important benefits to teens. o An acknowledgement that there nevertheless is a problem, and society has a collective responsibility to support teens. o Affirming that social media companies must play a central role in this effort and highlighting what they are doing to address concerns. - However, panelists did not think these rebuttals would be credible coming from Meta. The rebuttals would likely be perceived as even less convincing and in particular the "tools and autonomy" and "heightened concern" messages would be perceived as inappropriate or dismissive. Respondents believe they've been misled by Meta about teen wellbeing, and point to the whistleblower story that the company ignored its own research on the subject. - Though explored only in the UK, we found that parents and Insiders are willing to listen to independent research based on Meta's data. But there are concerns about Meta cherry-picking the data, so an independent researcher would need to vouch for the data and Meta's transparency during the process. Even if the data was analyzed by a third-party, independent researcher, respondents would wonder if Meta had provided them with all of the relevant data. They would want assurances from the researcher that this was the case before they take the conclusions seriously. - Recommendation: All audiences believed that social media is bad for teens, and these beliefs are too deeply-held and personal for us to change their minds. We found some elements of the rebuttals which, if combined, can turn down the volume of concern. To accomplish this, they would need to come from a respected third party, not from Meta itself. 43 PLTF EX-04350_0043 Landscape: Teen Wellbeing & Mental Health PLTF EX-04350_0044 Quant TL;DR PLTF EX-04350_0045 PLTF EX-04350_0046 Post Only Numeric Differences (US) Those who saw Tools and Autonomy are more likely than those who saw Scientific Rebuttal or Heightened Concern to rank social media as the most important contributor to teen well-being issues. [Data table - Outcome / Level / All / Group A-CSN Control / Group B-Scientific Rebuttal / Group C-Heightened Concern / Group D-Tools and Autonomy - Wellbeing Impact Problem Difference; Wellbeing Problem Ranking. Table values not legible in source extraction on this appendix page.] PLTF EX-04350_0047 Post Only Numeric Differences (UK) Respondents exposed to the CSN Control message were more likely than those exposed to Heightened Concern to rank social media as the biggest contributor to teen well-being issues. [Data table - Outcome / DV / Level / All / Group A-CSN Control / Group B-Scientific Rebuttal / Group C-Heightened Concern / Group D-Tools and Autonomy] Wellbeing Impact Problem Difference (1.1.2): The most important factor - All 32%, A 32%, B 32%, C 32%, D 34%; One of many/Not a factor - All 68%, A 68%, B 68%, C 68%, D 66% Wellbeing Problem Ranking (1.2.2): Rank 1 - All 53%, A 56%[sig C], B 55%, C 49%, D 53%; Rank 2 - All 17%, A 18%, B 14%, C 19%[sig B], D 19%[sig B]; Rank 3 - All 13%, A 12%, B 13%, C 13%, D 12%; Rank 4 - All 7%, A 6%, B 8%, C 8%, D 6%; Rank 5 - All 5%, A 4%, B 5%, C 7%, D 5%; Rank 6 - All 4%, A 3%, B 5%, C 4%, D 3% PLTF EX-04350_0048 Pre-Post Numeric Differences (US) Respondents exposed to the CSN Control Message perceive Meta to be more committed to teen well-being than respondents who saw any other message. [Duplicate of appendix chart - see page 35 for full data table: Wellbeing Impact Problem Difference, Wellbeing Problem Ranking Difference, Limit Difference, Good for World Difference, Committed Difference by Group A-D.] PLTF EX-04350_0049 Pre-Post Numeric Differences (UK) Mirroring the US findings, respondents in the CSN Control group have significantly higher means on perceptions of Facebook's commitment to teen well-being compared to all other messages. [Duplicate of appendix chart - see page 36 for full data table.] PLTF EX-04350_0050 Post Only Numeric Differences (US) Respondents who saw messages other than CSN Control were more likely to say that Meta is not committed or that they are unsure of Meta's commitment to teen well-being. [Duplicate of appendix chart - see page 39 for full data table.] PLTF EX-04350_0051 Post Only Numeric Differences (UK) Similar to the US, respondents who saw messages other than CSN Control are more likely to say that they are unsure or that Meta is not committed to teen well-being. [Duplicate of appendix chart - see page 41 for full data table.] PLTF EX-04350_0052 Pre-Post Numeric Differences (US) Respondents exposed to the CSN Control Message perceive Meta to be more committed to teen well-being than respondents who saw any other message. [Duplicate of appendix chart - see page 35 for full data table.] PLTF EX-04350_0053 Pre-Post Numeric Differences (UK) Mirroring the US findings, respondents in the CSN Control group have significantly higher means on perceptions of Facebook's commitment to teen well-being compared to all other messages. Outcome | DV | Mean | Group A – CSN Control | Group B – Scientific Rebuttal | Group C – Heightened Concern | Group D – Tools and Autonomy Wellbeing Impact Problem Difference | 1.1.1 | .05 | .06 | .03 | .05 | .07 | 1.1.2 | .05 | .05 | .02 | .04 | .06 Wellbeing Problem Ranking Difference | 1.2.2 | .19 | .29ᴮᶜ | .11 | .14 | .20 | 1.2.3 | .08 | .11ᴮ | .06 | .08 | .08 Limit Difference | 2.1.1 | .06 | .09ᶜ | .04 | -.02 | .11ᶜ | 2.1.2 | .05 | -.07 | -.05 | -.02 | .07ᶜ Good for World Difference | 2.2.1 | -.10 | -.10 | -.17 | -.03ᴮᴰ | -.12 | 2.2.2 | 0 | .01 | 0 | .02 | -.01 Committed Difference | 2.3.1 | .03 | .19ᴮᶜᴰ | -.10 | .03ᴮ | .01ᴮ | 2.3.2 | .05 | .13ᴮᶜᴰ | 0 | .04ᴮ | .02 Ns: Mean=1,882; Group A=465; Group B=463; Group C=483; Group D=471. Letters (A, B, C, D) indicate significance at 95% confidence in pairwise testing. DV explanation - https://docs.google.com/document/d/1ge1Wehk-xMsiaWEfNzI5EJYb_KHFWTlRz9o26xgOwNo/edit PLTF EX-04350_0054