Sayed Otaru: "The CI team has concluded a study (the numbers have not yet been formalized) and currently show that 500,000 IG underage accounts receive IIC on a weekly basis."
Chris ████: "I want to call your attention to the follow *non*-goal: "we will hold the line on IIC T2 prevalence on IG & Messenger, and respond to any spikes, but will not be goaling on prevalence reduction."
Malia ████: "Small correction to your understanding of the metric. The metric is ~500k victims per DAY in English markets only… So we expect the true situation is worse."
Yoav Shapira: "Child Safety is explicitly called out as a non-goal in our H2 plans. So if we can do something here, cool. But if we can do nothing at all, that's fine too."
████: "[D]id Core Dims confirm that their age models are optimized to catch age-lying minors and not age-lying adults? I thought it would be operational for both sets of age-liars."
████: "yep, definitely confirmed that their focus has been on misrepresenting minors and there has never been much focus on misrepresenting adults"
████: "i think this is largely an artifact of the history of product use cases for age models - facebook generally has gated minors out of experiences, so an adult lying about being a minor just gets gated out, no harm there. it is the integrity use case which is unique in that both kinds of misrepresentation are very important, and the lying adult may be much worse"
████: "And they are, in fact, bad at predicting age lying adults? B/c if the most relevant signal is the average age of your friend graph, age-lying adults could also be fairly predictable. Although, I know from experience that age-lying adults who do so to groom children create fake minor profiles for this explicit purpose and would usually have majority minor friends in their graph."
████ "not only are they bad at predicting them, they don't really know how bad they are because of a lack of high quality labelled data… but they know it's very bad"
████: "[O]ur age models are totally blind to this extraordinarily obvious exploit for the very abusive type we're trying to solve for"
████: "yeah, totally hear you on that b/c it's true: that fact pattern is a significant part of the highest severity harms, in addition to the IRL adult who is using messaging to abuse a minor he/she has access to or can persuade to meet up for IRL sex abuse. BUT the other part of the picture is the large numbers of adults who are quite brazen and honest about how old they are when reaching out to kids -- as demonstrated by this undercover investigation where a woman pretended to be an 11 year old and within days had 94 adult men reach out to her soliciting sexual contact"
████:"but this kind of case can and should be solved for by breaking unconnected adult<>minor contact"
████: "I think this is especially a problem on IG"
████: "it's a mitigation we have on messenger and want to get igd to parity"
████: "[W]hat specifically are we doing for child grooming (something I just heard about that is happening a lot on TikTok)?"
████: "Somewhere between zero and negligible. child safety is an explicit non-goal this half. I'd argue we're making it worse with Interop, but's that's a can of worms."
"Our policies only cover the explicit sexualization of minors. Recently we saw an uptick in hi-pri escalations showing trends of implicit sexualization of minors, particularly on IG. These escalations have received extensive media coverage in the US, India, South Korea, Japan, etc and has also kicked off discussions on further regulations."
"In March it was decided that we would not build a classifier to detect minors under 13 years old . . . We need leadership support to push product to act on knowledge we already have, i.e. clear our backlogs, disable verified u13s, build an u18 checkpoint to review minors who have been reported on 18+ products."
Child sexual exploitationFriend/Account SuggestionsDirect Messaging
"[W]ork is not planned to prevent FB and IG adults from initiating messages to IG minors. Limited country tests and data analyses have shown that ████ can demonstrably reduce CEI sharing but product leadership has been unwilling to prioritize these mitigations."
"We continue to advocate for strong discoverability and reachability controls in anticipation of Interop, including further restrictions on PYMK and GYSJ."
"Instagram Reels launching globally on August 5: Ahead of the launch, we have been reviewing curation guidelines, ensuring integrity measures are in place, and tracking cases of under / over-enforcement on Reels. Reels have also been integrated into the current IG HERO monitoring program which proactively reviews the most viral Reels for any harmful content. This program is currently operational in US, and expanding to India, Brazil, and Indonesia this week. This should mitigate the fact that our integrity classifiers are less effective on video and are not calibrated for short-form video in particular (i.e. all Reels content. One continued exposure for us is that we still do not have a sense of how minors' follow or DM graphs are changing after they go viral and the way in which this risk will be amplified when we launch globally. We had asked for this mitigation to be in place following the v2 launches but have hit a wall with the Data Science team working on this due to limited bandwidth. We are now reaching out to Messenger to see if they have bandwidth to help us measure and track this."
Sam Parker: "I checked in with PMM/comms and their POV is that the concerns have gotten stronger and they'd like to escalate.
Why they think H2 is a worse time to unship from an Outreach POV than their already strong concerns in early March for H1 unship:
• Time spent is a bigger concern due to COVID/spending more time online
• We just deprioritized the mental health team, so no new or upcoming features to point to here
• FB launched their v2 time spent tool on iOS in Q2 (Android coming in Q3) and got decent press around the re-launch
• Upcoming moments make the market environment sensitive in this area (suicide prevention day (sept), world mental health day (oct)) and there is concern that back-to-school will spark new issues in market perception due to the majority being online/remote learning so time spent online will likely be top-of-mind for many
How should we approach resolving the disagreement?"
████: "I'm curious who's the biggest advocate towards unshipping it now? I think the strongest reason is the data accuracy concerns so perhaps someone who can represent the risk there (DS/DE?). From a simplicity POV this doesn't actually add much clutter imo -- it's in the side menu, which feels like an appropriate place for a feature that's used but small % of DAP.
I think it would be helpful to do the following next steps: (1) Have a robust in-person discussion/review where advocates from both side of the spectrum are there to rep. (2) if we can't decide or just feel like it's good to raise awareness of this tradeoff, we can then tee up over email to IG Leads. I think we can use one of the upcoming WB reviews slots for (1)."
████: "Adding ████ to this thread to weigh in.
If H2 is a sensitive time to unship this feature, we could give your team an extension to unship next half. Either way, I think it's important for us to discuss its future sooner rather than later, so we know how we should proceed given that the feature in its current state is not a great user experience and could put us at risk of losing user trust."
Suicide & self-injuryChild sexual exploitationDirect Messaging
"Are we comfortable launching private-by-default for teens with the identified retention and engagement drops?"
"We want to make Instagram a safer space for teens. Instagram is behind the industry on keeping our platform safe for minors. Other platforms, including TikTok, have taken bold moves to reduce interactions by strangers. We previously announced private-by-default for minors as part of our December age collection moment which helped us land a favorable announcement. This is an area of high scrutiny by regulators and would be better for Instagram to address affirmatively, than forced by regulation.
Doing so will result in -2.2% teen DAP (-0.3% overall DAP), -1.9% teen TS (-0.3% overall), -2.3% teen feed impressions (-0.3% overall), -1.7% teen story impressions (-0.2% overall), -4.4% teen feed creation (-0.6% overall), -2.6% teen story creation (-0.4% overall) in 5 years, even after the Graph teams' mitigation work in H1. While small on top-line today, this will accrue over time as teens continue to join IG."
"Why should we default teens to private accounts?: Will Increase Teen Safety: Defaulting teens to private will help prevent high severity actions such as child grooming and inappropriate contact with minors. As a result of defaulting to private, private account adoption increases by 5.5x for teens, making it harder for bad actors to discover and message them."
"As a result of defaulting teens to private, we see decrease in retention and engagement which will compound over 5 years."
"Total launch impact is estimated to be -831K annual teen MAP drop from defaulting both primary and secondary teen accounts to private."
"We have previously announced this update in December as part of our age collection announcement and have discussed since then - reversing course will harm our credibility with press and external stakeholders."
"We have repeatedly heard from safety stakeholders that moving to private by default for teen accounts is the safest experience we can offer."
"You may recall that last year we temporarily removed some face altering AR effects from Instagram, following
questions raised about the potential impact of these effects on mental health. Removing these effects was an
interim measure that provided us time to reach out and collect a range of perspectives from creators, users,
and experts in psychology, research, body image activism, and augmented reality."
"Now, we'll once again allow people to create and share face-altering effects on Instagram, however we will not allow these effects to be discovered or promoted in our Effects Gallery (which is how many people find
effects)."
Kristin Shay Hendrix: "Totally. The teens findings were very, very concerning, though. So I want to do this right and not screw it up."
Kristin Shay Hendrix: "It was a pretty controversial/contentious Zuck review with Adam and Fidji not being in alignment."
Kristin Shay Hendrix: "The decision to test on both FB again was largely comms motivated."
Roberto Medri: "Schultz told me:'we proved Daisy does not improve well-being'"
Kristin Shay Hendrix: "Interesting Moira Burke in CDS has HUGELY pushed for Daisy to ship."
Roberto Medri: "What bothers me is everyone seems more motivated by incentives"
Roberto Medri: "CDS has the incentive 'publish an academic paper on this'"
Kristin Shay Hendrix: "We flagged this to Cox in our researcher Q&A with him how leads make decisions not based on evidence or people-first values."
Roberto Medri: "no finding, no paper"
Kristin Shay Hendrix: "We were in an awful position of Adam having publicly talked about this, all but promised we'd do this, and then trying to be as objective as possible."
Roberto Medri: "'I don't think this test will tell us much more than the previous one, and I don't think we have research staffed'"
Kristin Shay Hendrix: "And we were barely powered enough to detect the highly-active teens effect we did."
████: "At the review last week, we discussed that there is a very high bar to make a ship case for Daisy... the recommendation is to develop a comms plan to roll back Daisy."
████: "There are downsides to Daisy, but I think it is one of the clearest things (supported by research) that we can do to positively impact social comparison and well-being on IG and we should ship it. I'm amenable to ████ et al.'s recommendation, because I don't believe we will be able to produce sufficiently compelling evidence to overcome the opposition to Daisy in FB App... I'd also like to add that, if we can't ship Daisy given the external comms around it and the supporting research, then I am doubtful that we can ship any broad product changes with the purpose of improving user well-being."
Justin Cheng: "Social comparison may be mild for most people, but at the same time, we've shown (repeatedly) that it's among the top drivers of well-being on FB and IG. We can also show that social comparison does lead to decreased engagement via deactivation. Daisy is such a rare case where a product intervention can improve well-being for almost everyone that uses our products. So even if the benefit is small, especially in the likely case of not launching the 'pure' variant, I do hope that we do keep pushing on it."
Justin Cheng: "Separate from whether social comparison should be considered a driver or component of well-being, we know from previous internal and external research that social comparison is linked to multiple negative well-being outcomes (e.g., increased loneliness, worse body image, and negative mood or affect), as well as some positive outcomes (e.g., greater inspiration), so yes, I think it's useful as a measure on its own."
████: "The two Daisy-specific insights are compelling: 1) Negative social comparison decreased more over time for people who have Pure/Popular Daisy compared to people who do not. And 2) People with Pure or Popular Daisy tended to report less social comparison stemming from seeing content with a lot of Likes."
████: "you can build something smarter, than default
by private, for example not letting ppl who are 2+
degrees from you or some measure of "familiarity" get intouch with you"
Darius Kilstein: "This was never really about increasing safety… It was all for the pR wins"
████: "basically having 2 options with different levels of impact on metrics, wb, policy etc and let ████ decide based on the results"
████: "from our side we can only say we obviously want the one with less impact on metrics as a growth team"
████: "but i guess the better thing for metrics will be worse for wb/policy pov"
Darius Kilstein: "yep totally"
Darius Kilstein: "What they really want is to fix direct"
Darius Kilstein: "because strangers reach out"
Darius Kilstein: "so they should just do that"
Darius Kilstein: "thats what im trying to tell them"
Darius Kilstein: "stop creeps reaching out to kids in direct"
Darius Kilstein: "its simple"
"After opening the SparkAR product globally to third party creators, we've seen a rise in beauty related filters, both in creation and use… This has reinvigorated a conversation about our policies (both internal and external) on beauty effects and the stance that Facebook and Instagram should take on how those filters relate to body image, mental health, and wellbeing. This is especially sensitive for its potential impact on our younger users who are also the biggest demographic to use our Spark effects."
"Feedback Summary
- Generally, stakeholders support making a distinction between effects achieved via makeup versus those that can only be achieved via cosmetic surgery, though no line will be perfect.
- These extreme beauty effects can have severe impacts on both the individuals using the effects and those viewing the images.
- Children are particularly vulnerable, however many others are vulnerable as well: those with a history of mental health challenges, eating disorders, etc."
"We spoke with 18 experts including psychologists, researchers, body image activists, and AR professionals… A large majority, including individuals in the AR field, recommended prohibiting these filters, citing known impacts to body image and mental health of other forms of media that idealize unrealistic beauty standards."
████: "do you happen to know the status of the IG Smart Defaults launch?"
Vaishnavi Jayakumar: "I do – we were initially supposed to launch private by default on september 2 after approval from IG leads, but that has now been scrapped due to growth concerns, and the product team has been given some time to come up with ways to mitigate the drop in growth"
████: "[W]hat's your sense of the headwinds for bold social comparison interventions like Daisy, which may be associated with (not yet crystal clear) impacts on traditional metrics?"
Charles Sismondo: "The feedback we got in our IG leads review recognized this as a big problem that IG core mechanics makes worse, but one we should address. The difficulty in executing on a bold bet was recognized, with the comparison to Daisy also made. From a 'people first' perspective we have support to deal with the problem, but the headwinds will be the core mechanics itself and making sure we account for the impact on growth and engagement metrics..."
████: "Are there clear next steps for Daisy itself?"
Charles Sismondo: "last I heard, this was stuck on needing FB app to have a equivalent so there's a coherent approach across apps per Mark review."
"Ya I see this as two workstreams that are separate but related.
1. Keep regulators away, keep teens engaged
2. Make teens safe
I think where Growth is coming from is that for us, we only really have bandwidth for 1/."
Melissa ████: "hey yall, was chatting with Tomas about how many people are complaining about sexualized content getting recommended to them in Reels"
Melissa ████: "Some really yucky content. We're sexualizing underage girls, promoting some very iffy accounts, and serving this content to people who didn't ask for it instead of more inspiring interest content"
Thomas Kabbabe: "Thanks for flagging Melissa! I went to some of these accounts to check them out and they are FAR from being recommendable"
Melissa ████: "The Explore team fell into a trap where so much popular content in Explore was sexualized, it was really hard to ever make any adjustments to Explore to fix it because doing so would reduce time spent. Getting rid of it would impact communities like fitness, become really painful for our community because they were used to having that distribution which would create PR issues. So we should get ahead of this stuff before norms are established"
Sources include internal documents, public filings and press reports. “Alleged by plaintiffs” denotes the plaintiffs’ characterization, not Meta’s own words. Source labels identify provenance, not a finding of fact.
Meta & Children's Online Safety