Guy Rosen: "Should we consider limiting the CEI number we put in CSER to FB only, to avoid a step-change some time in the future when we lose access to private messaging? I know we've already released the top line number in the newsroom post -- so maybe it is already too late. But putting it in the CSER, where in the future we'll have a chart and everything, feels more irreversible."
Vishwanath ████: "This is based on an old snapshot of only 2 months of data, but going by the same rate, we'd have a smaller number around 3M.
Just looking at egregious CEI content almost 90% of it is in messenger.
I need to think more about this, but given we already shared it in the newsroom post, I think sharing a different number in CSER may be risky. There is very little chance of us not getting asked about it and then we basically out how much is deleted on messenger versus fb and that may just cause push back ('Why is Facebook now allowing millions of pieces of CEI to be sent on messenger?') later when we want to actually encrypt. It boils down to do we defend this now or do we defend this later.
Additionally, I think we delete terrorist propaganda also (the numbers there are smaller), and now co-ordinating harm (re: Myanmar) as well.
On safety carveouts: I think most of this (probably all) is images/videos (need to check) and if we can carve those out from encryption due to safety concerns, I think we could manage in the long run. Between privacy and safety, I think this could be a balancing act, if possible."
Guy Rosen: "Of the hard choices, I think excluding Messenger in CSER starting now (across the board) might be the less bad option. It seems defensible: yes, we published the CEI number including Messenger one-off, but generally the on-going report is FB only right now."
"Long Term Retention: The Young Ones are the Best Ones and Other Learnings"
"Our three major findings and associated implications are below."
1. "Tweens (approximate age 10-12) are special. People who join Facebook as tweens have the highest long term retention out of all age groups."
[a. Observations]:
i. "As an example, for people who joined Facebook in 2016, 12 year-olds (born in 2004) have long term retention (LTR) MAP_28 of 72% at an FB age of 365. 32 year-olds (born in 1984) have long term retention of 25% at the same FB age. This is a 3x difference for a 12 y/o versus a 32 y/o! The difference is similar in magnitude across multiple confirmation years."
iii. "Caveat: These findings rely on the Age Affinity model, and while it is the best we have for age, there are biases in the model which may influence the results."
[b. Implications]:
i. "ACTION: Given that tweens have long term retention multiple times (~3X) more than adults, understand why this is the case."
2. "For the Youth Team, prioritize tweens over all other age groups."
"We must onboard people into Facebook/Messenger as early as possible (tweens ideally) before they become attached to alternative social apps such as Snapchat, Instagram, and iMessage"
"Why we know this:
i. Teens who join Snapchat don't leave Snapchat after they grow up
ii. We expect the same for Instagram
iii. Your number #1 social app captures a vast majority of your time"
"What we need to do to help us achieve this:
Large numbers of tweens and teens, even in the US, still join and continue to use Facebook. Understanding why this is the case will help us replicate these conditions for more tweens/teens"
"Long Term Retention – Santosh and I wrapped up the first phase of the LTR work. Takeaways:
Tweens (approximate age 10-12) are special. People who join Facebook as tweens have the highest long term retention out of all age groups.
▪ THEREFORE, we should focus more on tweens
In a recent confirmation year, 2016, the higher the population penetration by year of birth, the lower the long term retention.
▪ THEREFORE, it is less straightforward that lifetime value increases as teen population penetration increases"
"Plans
• Teens
◦ Teen Decline Process - Partnering with ████████, who is leading the analysis
◦ Long term retention - Exploring new data set with ████
◦ Value of Teens - On hold because there isn't an urgent need. Additionally, long term retention findings call into question one of the assumptions
• U13
◦ Age-Up - Building a decision tree to agree on our assumptions and opportunities for the kids/tweens/teens strategy (████ leading)
◦ Tween Media/Apps – Brainstorming sources to find media/content most-watched and apps most used by tweens (████ leading)"
"We have definitively established tweens as the highest retention age group in the United States"
"To give you a sense of scale, people who joined Facebook at 11 years old have almost 4X the LTR of those who joined as 20 year-olds"
"If we want to win big with teens, we must bring them in as tweens"
"So what should we do?"
2. "Build for tweens (which may mean building a 'teen' product) above all other age categories, whether kids or late teens"
"On Instagram, teen adoption is more rapid year after year. There is no tween or teen problem on Instagram"
"Our original study had teens in it, but after the FB App Study made it to the press, we were asked to remove teens. We will keep teens in for Instagram, but not meter them, and we will beef up our consent. Facebook will not study teens as of now."
"There are concerns about publishing our final report, and even talking about this study externally. There were even fears of running this study and what will we do if we find out negative things about Facebook (comparisons to cigarette studies). We had to get this study reviewed all the way up to Nick and Javi."
Chris Norton: "As you might know, the Sunday Times will tomorrow run a (likely) front-page story with 30 families of suicide victims accusing Instagram of killing their children. The Government will write to us to tell us we need to do more and that this is a watershed moment. In my view it is.
This all stems from the BBC story earlier in the week which was very painful. The Instagram team have been handling and doing their best to communicate why we have the current set of policies. I understand that there is an XFN that is looking at those policies and trying to get experts to defend us.
That's all good, but this is getting to a critical level and the thing we lack at the moment is any action on our part. We are defending the status quo when the status quo is clearly unacceptable to media, many impacted families, and when revealed in press, will be unacceptable to the wider public.
At the least we need to say we are reviewing our specific policies in this area to make sure they are in the right place. I suspect Nick will get asked about this in his BBC interview on Monday as it is likely to continue to be significant news in the UK.
This is very likely Instagram's Lee Rigby moment, so I'm raising the flag. Being able to say we are reviewing won't help with coverage tomorrow, but it will when we get people coming to us for a response and it will for Nick's Monday media activity."
Nick Clegg: "+ Mark, Sheryl, Chris and Adam
Chris is not prone to hyperbole – and I agree with him, not just in a UK context but more widely, that our present policies and public stance on teenage self harm and suicide are so difficult to explain publicly that our current response looks convoluted and evasive.
This may be unfair - our policies are carefully designed to point youngsters who are sharing images of self-harm towards help – but we cannot afford to lose the confidence of the wider community, and press, on something like this.
The fact that we have age limits which are unenforced (unenforceable?) and that there are, as I understand it, important differences in the stringency of our policies on IG vs Blue App makes it difficult to claim we are doing all we can."
Monika Bickert: "In terms of improvements we can make, most are on the product side (see below), but those will take a while, especially because there is no IG PM for self injury."
Monika Bickert: "Yes, the policies are that we allow cries for help (often called admission or ideation) but don't allow celebration or encouragement of self-harm. It's a tough but very important line to maintain so people can get help. As you likely know, we've seen lives saved when they've been able to post cries for help on our services, and experts say that removing such posts could be the final straw that could cause someone to harm herself or himself."
Monika Bickert: "Just want you guys to see email I sent. PLEASE DO NOT FORWARD.
It struck me that the bottom line is that IG is violating our policies — not users! IG is choosing to display images in a violating way for any policy area where the caption matters. We have to build a system of labeling images as 'caption-dependent' for topics like hate, violence, bullying, SSI so that we can avoid search results."
████: "On that point, recommendation IG is looking into asap is leveraging content CO has identified as either admission or promotion, and using it as a signal to identify the related hashtags – then restricting those hashtags in one of two ways:
• For high confidence that hashtags are being used for this SSI content, the hashtag will be blocked (content will not appear when hashtag is searched).
• For lower confidence matching, show SSI resources to folks searching for that. Longer term we may be able to further refine this one to also only show some of the media."
Sheryl Sandberg: "If these are the right things to do, we should not need an article to push us to do them. We have reviewed our policies and enforcement in areas like this over and over with this in mind - and then we always find more to do. I am really alarmed by this.
We absolutely have to solve this problem. In obvious areas of concern (guns, opiots, harm, etc) we should be ahead of all of this.
Monika, Guy and John - schedule time with me when you are ready (but soon) so we can go through the roadmap and possible hot issues again. We need a new approach and another scrub. We should also review the red team - we should be able to find things before any journalist or kid on our platform.
And thanks for working on this over the weekend as well. I know a lot has gone into the improvements we have already made."
Guy Rosen: "We'll get together soon - the best way to review is to bring the integrity maturity framework from the last H1/H2 review, which shows our level of readiness across integrity problems and the prioritization of work.
While I'd love to say we have no open risks, there's enough at 'risk' maturity to keep me up at night (especially as we look across the family of apps) - and we should have our eyes wide open. I will keep banging the drum to remind us that we have to be focused on managing the fires so that we don't distract the long-term product work that will eventually prevent fires. The teams have caught up nicely over the past year, and we need to keep the momentum going into H1. We'll put this together and will talk soon."
Monika Bickert: "Nick, I just want to set expectations since I see that someone said Content Policy agreed to these changes. We are fine with them but they are mostly product changes, not policy changes, so execution is going to depend on product. (Our biggest contribution from a policy standpoint will be helping determine with hashtags they should block in search.) Guy's team will be leading the product stuff, and he is fabulous."
Nick Clegg: "Thx Monika - have been in touch with Guy separately who is keen to help.
I guess my central concern is that we don't have a simple, human answer to why IG carried harrowing self harm images that no reasonable person think should have appeared, and which seem to serve no self-help purpose whatsoever.
The outside world doesn't understand the difference between glorification and cries for help, still less hashtags or the difference between FB's policy and product work!
All I want is that we give reasonable answers - no one expects us to sort this out in 24 hours - with concrete and easy to understand actions."
Monika Bickert: "Yes; I don't mean you should say this externally. I am just setting expectations internally that my team can't execute on most of this stuff — it is about tech resources. We wouldn't say this externally either, but probably the two biggest answers to your question are these:
1. Search results on IG return sensitive topics AND give no captions
2. No proactive classifier running
There are others too, of course, but those are the big ones."
Chris Norton: "I'm sorry to share these images and I know your team do amazing work. But I'm really worried that the line we are drawing on what self harm content is allowed is in the wrong place. The father of Molly Russel is on the tv sofas tomorrow and they have shared new images with us, all of which CO say are allowed as they count as 'admission'. I know the rationale behind this, but we will lose the public on this as it just isn't what people want their children to be able to find.
I know we are reviewing the policies, but I wonder what else we can do to move the line in the interim.
This is causing us massive damage in the U.K."
Chris Norton: "We had yet another day where this led the BBC news (see attached email summary). That's a whole week of negative headlines, front pages and evening news. The policy rationale that we need to leave self-harm content up because it is a cry for help is not landing at all in the current environment. No third parties are coming out to support this position.
We come across as not caring and only interested in the money.
I believe we need a hand break moment where we say we will remove all self-harm images while we review our policies. If experts believe we should leave some of it up then they can make that case, and we can include their views in the review.
I'm sorry to be blunt but our current approach is doing immense damage to the Instagram brand and is making the case for draconian regulation."
Adam Mosseri: "One thing that is important for everyone to understand is that we cannot technically remove all SSI content from Instagram, both because the classifiers are far from 100% recall and because they're trained on violating content, which I'm sure is a smaller set of what the outside world will see as SSI. We are roughly a year away from being able to even consider this, and if we make this promise and don't deliver it'll set us back, not forward."
Nick Clegg: "We can disapprove of media storms but we can't ignore them, certainly not on this scale. Either the third party experts are not raising their voices in support of our approach, or the UK media is not prepared to listen to them. Either way, we appear in public to have no support for our stance."
Steve Hatch: "TLDR: We have seen a slowdown in spend immediately following the BBC report though to date no major advertiser has pulled all spend or taken a negative public position.However we remain in a very delicate position. All advertisers we have spoken toare taking a proceed with high caution 'wait and see' approach. They are expecting further concrete action and information from us and are not ruling out pulling spend particularly if there is further 'bad news' or no further progress made by us. They welcome the immediate steps but these are viewed as the beginning not a conclusionIn our conversations they understand the complexity of the area though many believe we aren't setting the line in the right place and that SSI content is far too easy to find."
"Advertisers are asking how we can allow content which we know contains sometimes graphic and distressing images to be 'adjacent to our adverts'. We do have Millward Brown research (which we haven't yet shared) that suggests that 9/10 users do not associate content with the content above or below it, this is research conducted with a small sample (6000 people) and 1 in 10 may still feel high risk without clients understanding the scale of the problem."
Nicola Mendelsohn: "Sheryl/Nick I know that you have both been involved in this but I'm escalating the situation in the UK to you following 10days of continuous negative feedback regarding Instagram, self harm, suicide and young people. Our messaging is not getting through and the actions that we have taken are not enough to stop the continuous negative drum beat. No experts will come out and speak on our behalf around the fact that we leave things up so that we can help people with self-harm tendencies. We have also now been informed that there will be an indepth Panorama programme next week that will look into broader issues around well being and under age use of our platform. This will create another round of negative stories. The situation with our clients is very fragile and I'm concerned that we are moving towards a client publicly pulling spend."
"I would urge us to think again about why we are allowing this imagery to stay up – it is truly horrific and our clients, especially those that are parents are finding it very difficult to square. I am also worried that this goes to the heart of defending the feed which we need to protect."
Sheryl Sandberg: "I want to understand if this is an enforcement issue or a police issue."
Body image & eating disordersAddiction & problematic useChild sexual exploitation
"NEGATIVE EXPERIENCES ON INSTAGRAM: SUMMARY OF KEY FINDINGS"
"Teens that feel pressure to look perfect, spending too much time, and fear of missing out often on IG are less likely to be satisfied with various dimensions of their well-being"
"These issues are also most prevalent on IG than any other social app"
"Specifically, the pressure to look perfect is significantly more common on Instagram than all other apps"
"While the general messaging of 'Connecting me to my friends' is the most preferred, 'Discovering new things to try' may be another effective messaging to address various issues"
[Frequency of Negative Experiences on Instagram (% Often + Very Often Among Teen IG MAP)]:
• Feeling like I have to look perfect: "32%"
• Feeling like I waste too much time: "23%"
• Feeling like I'm missing out on life: "19%"
• Seeing too much inappropriate content: "17%"
• Worrying about how my content will be used: "17%"
• Being contacted by strangers: "17%"
• Seeing my friends / followers argue too much: "16%"
"TEENS FEEL PRESSURE TO LOOK PERFECT ON INSTAGRAM MORE THAN ANY OTHER APP"
"Older teen girls in particular are most likely to experience this often on IG"
"They are also less likely to be satisfied with various dimensions of well-being"
"While 'Connecting me to my friends' is less a preferable messaging, there is no clear alternative"
"WHILE APP ADDICTION IS COMMON ON IG, IT ALSO OCCURS ON MANY OTHER APPS"
"Older teens girls are most likely to experience this issue and also have lower well-being scores in general"
"Beyond 'Connecting me to my friends' 'Helping me discover new things to try' and 'Connecting me with people who support me' are preferable messaging"
"However, SM has a negative impact when it magnifies their insecurities, e.g., comparing themselves to others physically and socially; teens also express feeling addicted to it and being bullied"
"Pressure to Look Perfect
'I feel I compare my life to others so if someone appears to have a good life I start feeling down about mine.' – Female 13-15"
"Inauthentic
'I believe that social media causes me to live life less in the moment and more digital. It makes me sad.' – Female 16-18"
"App Addiction
'I am addicted to it, it takes away time from other things I should be doing at times.' – Female 16-18"
"Fear of Missing Out
'It makes me see things I wish I had and see people I wish I could be like and I'm only me and it makes me sad because people have fun all the time and I don't.' – Male 13-15"
"Bullying
'A while back I got social media but I got bullied on it by my friends as they said mean things in a group chat but I couldn't do anything about it.' – Female 13-15"
"Q5b: Why do you say that social media has had [pipe: pQ5] on your life?
US: Teens who believe SM has a positive impact (n=743); Believe SM has had a negative impact (n=70)"
Karina Newton: "ps: i find it to be a very weird world we're living in where fb leadership is making threshold decisions of ig hashtags?"
████: "It's strange and unwelcome."
████: "So you're not alone."
████: "It's also the wrong level of granularity of discussion, imo."
████: "We keep pretending there's a purely technical solution."
████: "There's going to be SSI content on IG and FB for the near future. The existing classifiers are not hard to game."
████: "And (as you noted) false positive become a bigger concern as we adjust thresholds."
Karina Newton: "it's bananas"
████: "I'd almost rather go from 30% to >0% as opposed to gradual steps. At least that's easy to explain."
Karina Newton: "it doesn't seem like we are critically identifying what problems we're trying to solve for and what the actual business risk for those problems is. and by we i mean 'fb leadership'"
Karina Newton: "the biggest issue underlying all of this is less about the content but what is appropriate for teens and pre-teens and the elephant in the room there is age – I would be putting our efforts in addressing that core issue"
Karina Newton: "ie if we collected age on IG we could age-gate this content (assuming ssi classifiers worked) and if we used age classifiers we could detect under 13s and kick them off the platform so they wouldn't have access to content that's not appropriate for them to find)"
Karina Newton: "that's the actual policy on fire risk"
████: "It feels more like 'do we listen to experts or the media when they disagree?'"
Karina Newton: "who we listen to is critical"
████: "Makes sense for SSI. Drugs, CT..."
████: "Those are a bit different."
Karina Newton: "yeah, i hear you"
Karina Newton: "ty, glad to be living in bananas world with you two"
Max Eulenstein: "it's an exciting time :)"
Karina Newton: "yes, so many opportunities for growth :)"
Guy Rosen: "How are you holding up?"
Guy Rosen: "Sorry everything is blowing up :("
Monika Bickert: "I have some real concerns about how some stuff was handled by Nick, so that's not good. (E.g., him telling Sheryl in a meeting that our experts weren't willing to back up our policy, when in fact we had reached out to 3 experts as soon as we heard about the escalation and all 3 provided public quotes for attribution, and we had sent those to Nick and he had said he didn't want them.)"
Monika Bickert: "BUT on the bright side, I heard that at the Sheryl meeting, Cox and Schrep were really good and understood the policy"
Monika Bickert: "So overall, I'm fine"
Guy Rosen: "The SSI thing is hard and thrashy :("
Guy Rosen: "I saw Adam this morning, his takeaway from that is that we are not aligned with experts that much. De facto we are doing bad tihngs like leaving it up and discoverable and no one thinks that's good"
Monika Bickert: "Well, the discoverability thing is REALLY bad. There is no question that they need to fix that. But that's not our policies. They should be blocking the heck out of those hashtags."
Monika Bickert: "And the way IG displays stuff in search definitely violates our policies"
Monika Bickert: "I think none of us had realized that before"
Child sexual exploitationTerrorismDirect Messaging
Andrea Saul: "Hi Antigone! I'm going to start putting some thoughts together re: planning for encryption. While obviously real-world harm is the biggest risk, from a reputational standpoint, would you agree with these risks? Anything I'm missing?"
Andrea Saul: "1. CSER removal numbers will drastically go down, especially for CT and CEI
2. Referrals to LE will drastically decrease, especially for CT, CEI and grooming
1. For CT: 98% of our proactive LE referrals depend on Messenger content
3. Our ability to comply with LE requests will drastically decrease across the board. This is higher risk for CT, CEI, and grooming as well as other potential criminal cases
4. Groups like ISIS view this messaging system as the most useful for their core operation. Groomers have unfettered access to young people."
Antigone Davis: "These may be captured within what you have:
NCMEC reports will drop dramatically, these are reported to Congress, there will be policymaker follow up.
The quality and value of our reports to NCMEC will drop dramatically, rendering them close to useless.
FB allows pedophiles to find eachother and kids via social graph with easy transition to Messenger. WA does not make it easy to make social connections, meaning making Messenger e2ee will be far, far worse than anything we have seen/gotten a glimpse of on WA.
Our safety organizations and experts, especially NCMEC and Thorn, have helped us manage so many policymaker and press fires. I know there is this sense that they haven't been there for us but they have helped shorten our fires. (think NCII fire) and they have helped us land and defend things like Messenger Kids. We will lose them. They will abandon us and they will speak out against us. What we gain on the side of privacy we will lose on the side of safety. Honestly, I'd love if we did some hard core research to understand what we need to deliver on the privacy side to get our reputation back -- maybe there are other things we could do, most people don't even understand e2ee. Let's understand what they really want :)
On the diminished value of the NCMEC reports, this will hurt our rel with LE and NCMEC's rel with LE."
Andrea Saul: "Thank you so much. And yes, to sum up, this is going to suck."
Antigone Davis: "We will mange because we have to; maybe we will even find brilliant integrity solutions :)"
Andrea Saul: "Thats what I'm hoping for!"
Sources include internal documents, public filings and press reports. “Alleged by plaintiffs” denotes the plaintiffs’ characterization, not Meta’s own words. Source labels identify provenance, not a finding of fact.
Meta & Children's Online Safety