Shayli Roiz: "oh my gosh yall IG is a drug"
████: "Lol, I mean, all social media. We're basically pushers"
Shayli Roiz: "Seriously it is! We are causing Reward Deficit Disorder bc people are binging on IG so much they can't feel reward anymore…like their reward tolerance is so high."
Shayli Roiz: "I know Adam [Mosseri] doesn't want to hear it – he freaked out when I talked about dopamine in my teen fundamentals leads review but its undeniable!"
Shayli Roiz: "Its biological and psychological"
████: "[T]he top down directives drive it all towards making sure people keep coming back for more."
Shayli Roiz: "That would be fine if its productive but most of the time it isn't…the majority is just mindless scrolling and ads."
Darius Kilstein: btw there is a new netflix doco basically saying we're creating a world of addicts
████: Interesting. Yeah I was listening to an audio book yesterday about how people said they didn't like Facebook Newsfeed, just like how Netflix users say they want to save movies to watch for later. But people don't ever go back to watch their saved Netflix movies because they would rather binge more entertaining stuff, just like people did use Facebook more after we added Newsfeed.
████: Makes me feel like tech plays to humans' inability to have self control lol
Darius Kilstein: Yeh that's exactly what the doco says
Darius Kilstein: I think its true tbh
Darius Kilstein: Personally I think its most worrying for young people
Darius Kilstein: Like, as an adult, if you've got nothing better to do anyway, there's no major downside to spending time on social media
Darius Kilstein: but I do worry what it does to young people who are still developing their brains and social skills, as well as being more susceptable to mean comments or lack of friends/feedback
████: "I think there's a question of whether we're creating addicts or facilitating them. I read addiction is basically 50% genetic, 50% terrible coping skills, So is FB creating addicts or giving existing addicts a really accessible outlet?"
████: "a really accessible outlet that optimizes for time spent"
████: "which isn't inherently bad - it just keeps people coming back even when it stops being good for them"
████: "That's why I love things like 'You're all caught up for now' on feed for example - I feel like we have a responsibility to minimize the negative externality"
Darius Kilstein: "Yeh it's definitely facilitation"
Darius Kilstein: "Not creation"
████: "Have you guys read Hooked?"
Darius Kilstein: "I think though, without the right stimulus, somenone might never become an addict"
Darius Kilstein: "So it's a tricky one"
Darius Kilstein: "Its like, you'll never become a gambling addict if you don't visit vegas :p"
Darius Kilstein: "FYI in case anyone is interested, the documentary made me look into some data"
Darius Kilstein: "I found it interesting that 18% of US DAU are teens, but they make up more of the 5+ hour bucket"
Body image & eating disordersFriend/Account Suggestions
████: [Heavily Redacted Content]... "We're concerned as this means these accounts – which are clearly violating – will remain easy to find and may continue to be recommended – both to vulnerable people, and to journalists who are looking to write stories."
"We're concerned that our ability to find, remove and stop recommending these pro-ED accounts is going to be limited by #1 and #3, and we're going to keep getting negative press cycles about how we're failing to fulfil the promises we've made. It's also worth noting that the Molly Russell inquest will likely focus on the content we were recommending her. While that will be about suicide/self-harm/depressing content more than eating disorder content – showing we've made improvements to avoid recommending harmful content will be key."
████: "It would still be good to discuss blocking selected ED-related search terms for accounts. As others mentioned, it's hard to defend why we'd block certain terms in hashtag search and not in account search, and it's easy to find violating ED accounts in a couple of taps. When I search for 'thinspo' or 'proana' the accounts shown are overwhelmingly violating."
████: "Overall [the Telegraph reporter] thinks too much is being recommended across the board but acknowledged it's good certain terms can't be found in hashtag search [we also blocked some of the ones he flagged, which helped]. But 'tags' is the tab along so when you search proana [for example] loads of stuff comes up — the vast majority violating — already in the first two tabs [Top and Accounts], even if there's nothing in the 3 tab. He thinks that is us not delivering on the promise made by Adam in February 2019 that SSI content won't show up in search."
"He's holding us to the fact that Adam said we wouldn't show this stuff in search and yet, because of the disparity between hashtag search and account search that's hard to explain, it's incredibly easy to find. Worth noting he will also point to this stuff still being recommended in Explore and suggested accounts in feed too"
████: On explore and recommendations in feed we're exposed but have made the changes promised and are now behind due to classifiers + Covid. On search we're exposed with nowhere to hide. We're not necessarily saying blocking users in this scenario — but stopping anything from appearing in search when you type things like proana, thinspo etc. [i.e. anything that could be inherently violating]."
████: "Reward is the satisfaction one feels when they achieve the desired outcome from actions they have taken to fulfill a need. This means, when an app does the job it was hired to do consistently, quickly and effectively, usage of the app itself is perceived as a rewarding experience overall by users."
"The ratio of time spent to reward reached was perceived as unsatisfactory on Instagram. Participants did not feel rewarded as often or as deeply on Instagram in comparison to competitor apps."
"When reward was reached less often or slower than expected, people described opening the app habitually and using it passively instead of intently. This was the case for many of our participants regarding Instagram."
████: "I've done some research about the effect of extrinsic rewards on intrinsic motivations and it does point to obtaining short-term rewards (in our case, lightweight interactions like reactions, emojis, and likes) as potentially replacing intrinsic motivations which creates a perpetual ecosystem of superficial connections. It could be quite a lift to shift users' mental models of how to use IG, but as you noted there could be potential to reintroduce IG (or maybe Threads?) to drive relationship building when we know that is their intent for their secondary acct."
"Sex trafficking classifier: We have one engineer dedicated to this work and we have no data science support to do some basic understand work to lower the strikes threshold for prostitution and sexual solicitation strikes (currently set at 17x, which is too high). This area is severely under-resourced given our SESTA-FOSTA liability."
"[T]here's also a current u13 backlog (~450K job backlog) that must be addressed via workload allocation and automation (i.e. auto-accept all other age change requests, which is a combo of people going from 13-18 > 18 and people going from 18 to 13- 18). This backlog is currently not resourced and has been escalated to Ops Resources, In the current state, we would not be able to reduce the backlog to zero for another year. The cross-enforcement of u13s may also be at risk if the 2020 elections need more investment."
"Immediate Product Vulnerabilities
- We lack a U13 classifier, insufficient means to detect and remove ul3s at scale
- We also have deprioritized U13 enforcement because COPPA compliance is at risk when we sit on knowledge of u13, incentivizes not knowing
- Age prediction models are not that accurate, reducing the efficacy of classifiers like the grooming classifier and requiring more reliance on human review
- Age prediction models are not globally operational; as a result, integrity measures that leverage age are inconsistent and unreliable, including child safety protections
- Age prediction models are not globally resilient, they tend to perform better in Western markets and worse in emerging markets
- Age models do not work as well on IG; as a result, we have no ability to more granularly gate minors out from age-related experiences on Instagram
- As a result of all of the above, are age-gating doesn't work as well as it should yet we offer age-gated products like Watch Together, FB Dating,... and we have minors who appear as young as 12 years old.
- Lack of u18 checkpoint: Risk having u18 users on 18+ products like FB Dating, Marketplace, Mentorship, or as under-age parents on Messenger Kids
- Age verification: We remain reactive on age verification, pushing for a device side solution, when we're getting increasing policymaker pressure to be proactive while our competitors improve their approaches"
"Overall we are quite vulnerable when it comes to age management on our apps. To some degree that is because leadership has decided to live with certain risks at least in the near term. For example in March, we decided we would not build a classifier to detect minors under 13 years old. Identity Integrity identified a PM who will own age verification but we will only prioritize this work when we face clear legal obligations to do so."
"Identity Integrity did agree to enable cross-enforcement of ul3s on IG who have hard-linked FB accounts since basic COPPA compliance is at risk when product does not prioritize checkpointing and disabling u13s, There is a chance, however, that this fails to get resourced because of engineering constraints on that team."
"Age models do not work as well on IG because 1G only started to collect age data in December 2019, In addition, Adam Mosseri will not support upselling age collection for existing users because he sees it as intrusive data collection."
"In March it was decided that we would not build a classifier to detect minors under 13 years old… We need leadership support to push product to act on knowledge we already have, i.e. clear our backlogs, disable verified u13s"
Body image & eating disordersFiltersContent Algorithm
"Immediate Product vulnerabilities
- Some core features of products connect to larger societal challenges around body image and objectification of women (e.g., AR face altering filters on Instagram, likes, etc.)
- Prominence of eating disorder-related content in searches and recommendation surfaces like Explore because of both (a) limited policy definitions around what constitutes ED content and (b) no ED-related classifiers that we can run across our surfaces."
Child sexual exploitationFriend/Account SuggestionsDirect Messaging
Our biggest enforcement gap in relation to age for child safety is currently our inability to automate decisions accurately to enforce against groomers. The Adult Classifier — while it is our best available age model to determine whether someone is a teen or non-teen — cannot determine age gaps at a granular level, which hampers our ability to action our policies, which rely on making specific age determinations, i.e. if the victim is 14 years old or younger and the offender is 18, disable and escalate, but if the victim is 15 and the offender is 18, checkpoint the offender. However, single-year granularity is really difficult to achieve with machine learning models. Thus, we'd also need additional human reviewer investment to get better at this.
Body image & eating disordersAddiction & problematic useContent AlgorithmInfinite Scroll+1
"Are we making problematic usage worse on Instagram?"
"Teens report that it's easy to get distracted and mindlessly scroll in the app without getting a lot of meaning/utility"
"Feeding the Spiral of Social Comparison"
"When someone looks at one post that makes them feel bad, IG mechanics nudge them to keep consuming similar content"
"Makes problems worse: Can exacerbate existing issues e.g. eating disorders and body image issues"
"Exposure to problematic content: Teens can unintentionally end up consuming content that makes them feel worse, including SSI content"
"Some dimensions of IG experience can prompt consuming similar type of content for a period of time, making teens who already feel bad, feel worse"
"In this note, we find that selfie-taking can make people feel worse about themselves. Sharing more selfies was associated with feeling more negative social comparison. In Stories, seeing or sharing more selfies that had filters or effects applied made people feel even worse."
"Goal: We will goal on delivering disproportionate distribution outcomes relative to author follow count. In other words, creating moments where people get 10-100x plays relative to their following when they create great Reels. This is in service of our Producer PR goal."
"Focus: We will focus on small accounts (<1k followers), experimenting (1k-10k), and aspiring creators (10k-200k). The incentive should be to get more content from smaller creators but keep an eye on how this affects larger ones. Driving creation with them is the only way to hit our 5% producer PR goal."
"A qualitative look at the predicted civic Reels getting the widest unconnected distribution (where the viewer does not follow the author) surfaced several disturbing trends. In particular, this (small) sample suggests we are recommending content from users that regularly post likely misinfo, spam, or information related to conspiracy theories. The vast majority of these top predicted civic Reels were also from the right and far right, suggesting we may be amplifying certain viewpoints over others."
"We seemed to be amplifying extreme voices, and may also be disproportionately amplifying *one side* of the ideological spectrum"
"Amplifying small accounts is an extremely risky thing to do right before an election!"
"The wide distribution given to small accounts likely compounds existing integrity vulnerabilities on Reels, because we expect most of our integrity classifiers to perform worse on this type of content (since most weren't trained on this type of media)."
"Hashtags are built by the community and the community has a voice. If we start filtering media which is non-rec, we are suppressing voice. We should NOT filter non-recommendable content from Hashtag pages."
"Hashtags are a source of many escalations, since the policy on hashtags is not well understood, and frankly is confusing even to us. Unfortunately, there are forces that use Hashtags in a harmful way and we do not have a consistent, well-socialized way to approach Integrity challenges on Hashtags. Being a complex object, hashtags has many touchpoints that across different surfaces and products and IG that each have their own
nuances. Hence, it is all the more important to have a consistent, well- socialized policy/protocol/enforcement strategy for each of these touch- points to holistically tackle integrity challenges on Hashtags."
████: "My takeaway was that if we want product leadership to prioritize wellbeing product changes we have to come up with a killer statistic. (That seems both simple and hard.)"
████: "and for some problems, probably impossible"
████" @Justin, I think some of our descriptive stats from the body image survey could potentially be enough to change the conversation"
████: "was this 'girl teens being 32x more likely to get unsolicited sex talk on IG compared to FB' the main example?"
████: "i think so"
████: "I think this was the number of teens who had a DM/Messenger thread with a 'sketchy' adult. So on IG there were 32x as many of these threads in the US than on FB."
████: "I think it's sex-talk between adult and minor, from a prior conversation I had with Nino"
████: "I had some convesations with the child safety team when I was working on age affinity. It's low prevalence, but extremely high intensity/risk/damaging"
████: "The investigator I talked to at FB was the saddest person I've ever met. Seriously, it was heart breaking."
████: "Not that it's not important, but I guess my point is that there's a lot of leeway in how stats are presented to achieve some goal."
████: "more than #fake it's like #lyingwithstatistics"
████: "I think the point was that there was an effort at FB, not a lot of attention at IG, and when they looked they realized it was much worse on this side of the fence"
Antigone Davis: "[O]n the request for us to improve our age models, what was the decision leadership took on this when they decided to move age verification efforts to device side?"
████: "[T]here was no explicit decision to improving our existing age models. The leadership decision was limited to NOT building an u13 classifier"
████: "According to Karina, Adam is not on board with prompting to collect age from existing users (unless there's a clear product need)."
Miki Rothschild: "I just watched the social dilema yesterday and it made me sad b/c while I don't agree with a lot of what they say there's some truth to it. Have you watched it?"
████: "Yes the day it came out… I agree with the core points… like there seems to be a problem with our business model and definitely a problem with our metrics… We really need to take a very close look at what sessions (metric) is doing to vulnerable sub
populations for example: teens, people with mental health issues, sensitive geographies."
Miki Rothschild: "Yes"
████: "Within feed, we are attempting to operationalize a sentiment based metric… but survey based metrics don't have a level playing ground with behavioral metrics like sessions and MSI."
"We seek to accelerate our ability to address content issues on platform for topics such as eating disorders, for which we have limited mitigations"
"Hashtags that promote eating disorders are relatively easy to find, resulting in ED content."
"Facebook is perceived to be a contributor to mental health issues and … on IG particularly, body image issues."
Antigone Davis: "Hey did you ask for more child safety specific HC? If you have 5 minutes for a call, I can provide some context for the question."
Christopher ████: "would you benefit from specific examples of where we fall short?
eg:
- we do nothing on complex objects (notably clearly violating groups & pages)
- we can't detect 'new' CEI
- we're scrabbling to be ready for encryption
- our IIC numbers (notably on IG) are through the roof
- we don't have comprehensive logging or measurement, we need both
- anything beyond *sexual* abuse is effectively being ignored
- we don't have bandwidth to properly support new products (i worry about Reels & RTC in particular)
- we have big surface gaps, reporting gaps, detection gaps, and review capacity gaps"
"Starting in 2018, there was growing concern in the public (amongst media, academics, policy elites, and everyday users) that technology companies were deliberately manipulating users by designing addictive products that can harm well-being and subvert control, labeling these concerns 'Facebook Addiction'. Efforts were spun up across Facebook and Instagram to more deeply understand these concerns. Initial products were launched to give people more control over their use, though efforts were defunded end of 2019 with additional opportunities still available."
"We internally aligned on 'Problematic Use' (PU) as a more accurate term to describe habitual or undesired use of Facebook."
"A person is classified as experiencing problematic use through self-reported survey measures. Internally, we define problematic use as experiencing both of the following issues either 'very often' or 'all the time': Lack of control or feelings of guilt over Facebook use. Negative impact in at least one of the following areas: sleep, parenting, social relationships, or productivity."
"12.5% of people worldwide experience problematic use."
"Triggers for Problematic Use on Facebook"
"Notifications - Getting too many minor/irrelevant notifications. Try to only look at important ones but get sucked into longer sessions."
"Mystery of the algorithm - Uncertainty over if they will see posts from those they want; if they can find a post again later."
"Videos - Easy to get immersed (especially before bedtime). Auto-play exacerbates the issue."
"Fear of missing out (FOMO) - Worry about missing important world news or updates in their social circles."
"Groups - Get exponentially more notifications, engaged in chat threads."
"There is a graduation in use of apps as kids age. YouTube is a constant but kids tend to move from 'play' and single-purpose apps at age 6-9 through to more social apps such as TikTok, Facebook and Instagram from ages 10-12."
"Messenger Kids is more associated with these 'single purpose' apps than the more social apps and begins to drop off at 10-12."
"There is no one single 'winning' app that kids can use to contact all their friends."
"However, taken as a whole, the majority of kids are using at least one of the Facebook family of apps to keep in touch with most of their friends."
Calvin ████: "Melissa was after me"
Tomas Kabbabe: "superstar. but decided to go the manager route too"
Calvin ████: "Yeah I hear her during the quarterly"
Tomas Kabbabe: "she killed daisy dude"
Calvin ████: "Daisy?"
Tomas Kabbabe: "Oh that's right. You're in WhatsApp. Daisy is that project that hides likes"
Calvin ████: "Ahhhh"
Tomas Kabbabe: "It's been a monumental nightmare for over a year"
Calvin ████: "That's great for teens"
Tomas Kabbabe: "Adam got us into a huge hole for announcing it publicly"
Tomas Kabbabe: "And the test was a disaster"
Tomas Kabbabe: "And people felt reeeeeaaaaaally passionately about Daisy one way or another. It had to be killed"
Tomas Kabbabe: "Melissa infiltrated CDS and Research"
Calvin ████: "Oh shit. It got killed?"
Tomas Kabbabe: "Well, because of Adam getting us into a hole, it can't be fully killed. So, we'll create a 'Daisy mode.' Where you can opt in"
Calvin ████: "But expect no one will use it then"
Tomas Kabbabe: "exactly. This is what's funny. Everyone said that they loved it BC it's the virtuous thing to say"
Tomas Kabbabe: "But the tests were disastrous on engagement. And even sentiment was unmoved"
Tomas Kabbabe: "People would claim shit like 'teens in indonesia and Android on the 2018 cohort are positive on posting.' Just looking for excuses to ship"
Calvin ████: "Yeah the 5 people we talked to in qual research (biasedly selected) loved it"
"I'm a huge fan of unshipping things. I use unship review as a recruiter message to potential Instagrammers.
And I'd love to replace this feature with something better. I've asked to unship it in the past.
But I don't think we can touch it for months, maybe even more. The regulatory and brand risk from removing our only addiction-related feature outweighs (imo) the wins around user trust in the data from the few users who use it."
Child sexual exploitationFriend/Account Suggestions
"Context: In August 2020, we found out that there are ~38x as many IIC victims on IGDM as on FB Messenger (Aug 2020). Despite this, CEI deletes on IG are ~21x lower and IIC auto-enforcing on IG is 90x lower compared to on FB."
"Actors take advantage of our tools on Instagram to find and inappropriately engage with children. While we have made strides in tackling the worst of the worst, from CEI to IIC, Instagram has a long journey ahead in understanding how these interaction play out on Instagram and how bad actors use our systems and products to do harm, especially on newer and unique surfaces to IG."
"Here are the slides Antigone has put together. There is some good material here on the scale of the internal problem illustrating why we're just not set up adequately to answer the challenge. I think she could both unpack some of those problems (slides aren't particularly conducive to the level of detail that's needed here), and also expand on the section near the beginning to emphasize the cumulative regulatory pressures we're under, and why they create both inevitability and urgency."
"Child safety enables policymakers and advocates to focus the public on visceral issues and ignore complexities. We need the complexities to be part of the discussion if we hope to move toward reasonable legislation across important areas like content regulation, age of consent, and user privacy."
"Addressing child safety in a more robust way will NOT clear away legislative efforts in these areas, but it WILL enable us to guide legislation toward models that align more closely with our legislative and business goals (e.g., content regulatory model that looks more like what has been proposed by CCIA than what we are seeing w/EARN IT or Cornyn-Machin or worst versions of UK Online Harms)."
Sources include internal documents, public filings and press reports. “Alleged by plaintiffs” denotes the plaintiffs’ characterization, not Meta’s own words. Source labels identify provenance, not a finding of fact.
Meta & Children's Online Safety