Child sexual exploitationBullying & harassmentDrugsContent Algorithm
"Overall, while IG Reels has generally implemented the SOX controls required for ad products, several areas in content integrity and product features need significant improvements. IA identified 4 High risk, 20 Medium risk, and 3 Low risk observations during the audit. Based on the overall health of the processes, an 'Orange' rating was assigned to this report."
"Insufficient reduction of interactions between minors and potential child safety violators."
"Private Reels Surfaced to Unconnected Users
The IG Reels Product team has not documented the decision and mitigations related to the logging and monitoring of private Reels viewed by unconnected users. As a result, privacy risks associated with user settings may not be identified timely or enforced appropriately.
For example, on a sample date, after considering the 12-hour caching time, 1.4% of VPVs on private Reels are viewed by unconnected users. 3.75% of private accounts with Reels VPVs were impacted."
"The IG Reels Product team has not documented the decision and mitigations related to the logging and monitoring of private Reels viewed by unconnected users. As a result, privacy risks associated with user settings may not be identified timely or enforced appropriately."
"Potential Violators Able to Interact with Unconnected Minors
The IG Wellbeing team does not sufficiently enforce bi-directional filtering between minors and potential violators on recommendation surfaces. As a result, the potential violators may bring harm to minors through the Reels surface.
On a sampled day tested,
1) Minors and potential violators can view and reply to comments from each other on a third user's Reels.
2) Minors can see and view Reels created by potential violators on the unconnected surfaces (i.e., Explore and Reels tab) and vice versa. Out of all VPVs made by potential violators, 12.23% were on unconnected minors (i.e., not friend, follower, or followee). The percentage is 0.37% vice versa."
"The IG Relevance Integrity team does not adequately identify or monitor trends in Reels. As a result, potentially harmful Reels trends may get wide distribution, and the teams may not be able to intervene timely."
"The IG Relevance Integrity team does not monitor the efficacy (e.g., False Positive or Negative Rates) of the demotion model for objectionable content (e.g., regulated good, drugs, and violence). As a result, deteriorating model performance over time may go undetected."
"The IG Relevance Integrity team does not regularly monitor the importance of the inputs to the objectionable content demotion model (e.g., regulated good, drugs, and violence) to assess its reasonableness. As a result, significant changes in the importance of inputs may go undetected, leading to degraded model performance."
"IG Relevance Integrity team has not planned for human intervention when the scheduled daily model training fails consecutively for a few days. As a result, the model may not be trained with the latest production data leading to unsatisfactory performance.
For example, in July 2021, only 7 days out of 31 days had successful training. For the other days, the training data table was blacklisted due to upstream data availability issues that were not resolved."
"The IG Reels Engineering team does not monitor the percentage of exceptions in blocking Tags and @Mentions based on user settings. As a result, the team may not be able to act timely when the user settings are not honored."
"The IG wellbeing team currently does not have standard governance processes for IG Reels comment bullying and offensive audio name models. Specifically, the following are missing:
1. Monitoring of anomalies for important features
2. Monitoring of model performance
3. Training, refreshing, and deprecating of production model
As a result, non-maintenance of IG well-being bullying production ML models may result in models not performing as expected, which may cause bad comments not filtered/covered properly."
Elizabeth Crenshaw: "[W]e're going to create a little infographic/timeline of youth safety efforts… and perhaps just give small heads up to a few reporters but nothing major"
████: "id be slightly nervous about alerting reporters to the fact that these privacy settings were defaulted off before 🙁"
████: "also – it occurred to me the other day – we say 'there are lots of valid reasons why young people woudl have public accounts', and cite the example of creators/young entrepreneurs… but wouldnt they have a creator/biz accounts, which are automatically public?"
Elizabeth Crenshaw: "yes… but still valid… i think???"
████: just feels like we're then making the rules for those exceptions, versus deciding to err on safety and just say under 16 accounts on IG are private… the risk could then be if u16s then switched to professional accounts so they could have public accounts – but im not sure how big a risk that is"
Elizabeth Crenshaw: "i dont disagree… but you know how contentious this change alone was"
████: "it's absolutely mad that it was so contentious… with adam or others too?... we will never get out of this mess if he/we're not just prepared to ERR ON THE SIDE OF SAFETY… jesus… i know i'm advocating for NOT letting them switch back – but i don't get why defaulting into private was contentious, given they CAN just switch back!"
Elizabeth Crenshaw: "it was mostly adam for sure… it was a huge growth hit… like shockingly so"
████: "would he want any tom dick or harry being able to see all his kids' content, follow them etc? is he fucking nuts?... interesting… well not doing it for that reason would have been unforgivable so im glad we did!
Elizabeth Crenshaw: "i mean we did NOT do it for a very long period of time because of that… i agree though"
"My daughter ████, now 16, is what we call an experimenting creator. She posts about cars, and in a recent post, someone commented, 'Get back to the kitchen.' It was an awful experience for her -- she tried to report it, to no avail, as this is something far from violating policy. So, her only options were to delete or block. Why does someone think it is ok to make that comment? I believe it is because it doesn't violate policy, and other than comment warning, deleting, or blocking, there is no feature that helps people know that kind of behavior is not ok."
"████ often gets unwanted sexual advances in messages, like 'friends with benefits?' from other teens and friends of friends. This is a negative experience for her, but it's not always clear how to express this on the reporting flow... When people come into someone's DMs, maybe we should consider a 'my house, my rules' approach: It doesn't matter what people say -- if it feels like a sexual advance in my space, it is."
"Everyone in the industry has the same problems right now. Prevalence-based policy measures, while necessary, don't create a safe and supportive community alone; you're always behind the latest harmful thing."
Justin Cheng: "It will be hard to optimize for intentional time spent without allowing for regressions in time spent. Is this okay?"
████: "[W]e think it is important to call out the tension in core priorities around defending time spent market cap, particularly for teens, and making meaningful progress on 'intentional time spent.' We would love to get feedback and direction on how we should approach these tradeoffs. Our POV is where he have identified and have strong conviction on instances of problematic/harmful time spent, then we should be ready to incur the engagement regressions associated with addressing them."
"Past research on problematic use on Facebook, as well as prior academic research suggests the existence of a U-shaped curve, where people with a moderate number of sessions report the lowest rate of problematic use. As session count increases, more and more people report feeling like they spend a lot of time on Facebook or report sleep problems."
"NAC prevalence increases with time spent overall, particularly among women."
"HLM also suggests that among teens, body image, social comparison and particularly sleep problems are more intense, have higher reach, and/or have more teens saying that IG makes it worse. So arguably we should still be prioritizing these 3 things over problematic use."
"Circling back re: investment needed to strengthen Meta's position on well-being (see original email from Aug below).
This investment is important to ensure we have the product roadmaps necessary to stand behind our external narrative of well-being on our apps, and soon in the metaverse."
"Given the increased urgency of all this since the initial escalation in August, are you supportive of funding this scaled back proposal?"
Suicide & self-injuryBullying & harassmentChild sexual exploitation
"What makes research sensitive?
Sensitive means research that may have comms, policy, or legal risks"
"Sensitive Populations
• U13 (under 13) children
• U18 (under 18) teens
• specific race/ethnicity
• political/economic group
• health or disability traits
• sexual orientation, gender identity"
"Sensitive Topics
• elections
• alcohol / drugs / tobacco
• privacy settings
• accessibility
• health and safety
• integrity (e.g. online abuse, suicide, harassment, bullying, mental health, trafficking, CEI)"
"Sensitive Data Types
• biometric data, face recognition, voice recognition
• geolocation data
• employee data
• third party data
• off-platform tracking
• user-generated content (UGC)
• WhatsApp data"
"For sensitive research, we can mitigate the risk in two ways:
• Be mindful about how we structure the research and communicate the findings
• Conduct highly-sensitive research under attorney-client privilege"
"When framing sensitive research findings, DON'T:
• Avoid vague terms or overgeneralizing research results
• Avoid use of legal terminology or making legal conclusions
• Avoid clickbait or sensationalized statements or claims
◦ Be mindful of buzzwords or triggering phrases
◦ Stick to the objective facts"
"Announcing this after my testify is worse than before, and we talked this. It will leak, and it will make it look like I was hiding something. It also will make claims I make about research in testimony less credible."
" If we know that a user is underage and they've also explicitly admitted to owning several accounts by hardlinking them via the account centre, we must enforce on all of them, not just the account that's been reviewed. ## Problem As a result of major infra changes some time ago and reoccurring issues related to alignment of FB and IG deletion policy we haven't had FB→IG enforcement propagation working consistently for over a year, and we've never had IG→FB enforcement propagation... We're inactivating around 30k extra Instagram accounts per week as a result of propagations from FB! That's 70% more than we were inactivating prior to propagation... This remains an important part of maturing our enforcement mechanisms and having defendable efforts - because these are accounts that we know to be owned by underage users."
"Thanks for the feedback! Team will keep meaningful adoption and leave as a % but does not believe we have a path to hitting that goal so would like to request we reduce from 0.6% to 0.25% given the tactics we have aligned and prioritized for this half."
"Revised Summary of recommended H122 Numerical goal changes for IG Mental Well-being:
Original goal: 0.6% weekly active teens with meaningful adoption of mental well-being features (0.843M/...
V2 Proposal: 0.26% weekly active teens meaningfully adopting Take a Break or Alt Topic Nudge (0.58M/225M)"
"Team was unable to test Take a Break opt-out by default for new teens due to 'one-way door' principle, but it was used as a high confidence bet in goal planning contributing >80% of teams expected impact for the half"
"0.26% weekly active teens meaningfully adopting TaB or Alt Topic is seen as a hittable goal, but this will still push the team given the time left in the quarter and other goals the team is working to hit outside of adoption."
We currently preserve underage data for up to 180d after being disabled - Youth pXFN have warned us that this is not defensible to regulators (particularly COPPA)
We lack the ability to include any messages or posts where users self-admit to being U13 in our enforcement jobs at the moment. This can lead to cases where users who have explicitly publically admitted to being underage are still not enforced upon after being reported.
We have no clear strategy on our AAM violation - what actions should we take when we detect them? What sort of proactive detection are we seeking to implement? Do we have a quantified level of danger these users pose?
We don't have visibility on harm associated with Adult/Minor age misrepresention, having this will allow us to measure effectiveness of enforcements in this domain.
"Monitoring SLA and job inventories (enforcement, appeals) / Taking actions to prevent SLA dropping, backlogs building, or jobs expiring: Maintaining our commitments to enforce underage policies and unblock falsely checkpointed accounts within a reasonable timeframe (48 hours). Open Questions: 1. Are the new teams going to care about the SLA? Should we still be caring about the SLA? 2. Should we set a deadline where we will no longer be acting to remedy any ops capacity issues?"
"Deprecation of dormant checkpoint policy: Deprecation this indefensible policy as part of Roadmap to BASIC maturity."
"Ensure legislation in Spain and South Korea is being applied properly: We enforce this correctly in most places - but we don't have proper testing to guarantee this, and without a proper audit we are at risk of enforcement (or communication) gaps. The age limit in Spain and South Korea is 14, rather than 13 in the rest of the world."
"Reduce U13 retention window to 60d: Youth pXFN have advised us that it is not defensible to retain underage user data for 180d (the standard for admin scheduled deletions) - we need to coordinate with Deletion API and modify propagation behaviour to reduce this to 60d"
"Understand and remove (or pXFN align) soft-linking enforcement against underage accounts for FB: We uncovered the use of soft-linking to disable FB accounts for being underage. We are unsure why and want to understand what this is being triggered by. We have no ground truth for what an underage account is, so we cannot measure the precision of an approach where we remove soft-linked accounts. In the current situation, this approach may be indefensible"
"Update DOB to the DOB of the submitted ID when a user successfully appeals the underage checkpoint: On IG we don't update DOB when appeal is successful, on FB we do but we believe it may be broken (as updating DOB is blocked when account is restricted). By failing to update account DOBs after they have been ID verified, we are facilitating age misrepresentation which could have harmful downstream effects."
"Restrict Adult Classifier enforcement automation policy to supported regions: One of the automations we run on reports of users being underage relies on the adult classifier model from core dimensions. At the moment on Instagram we are calling this model for users in unsupported regions and actioning them based on their stated age instead of realising they have no prediction and skipping this automation policy for them. Ignoring reports of accounts being underage based on their stated age is a poor approach and means we are likely ignoring reports of many legitimately underage accounts."
"Add the ability to include content from messages or posts in our underage enforcement review jobs: At the moment our third party report review doesn't include any message or post content from users, so totally lacks any insight into place where users have admitted to being underage. Our review protocol is currently based almost exclusively on account photos. If the account in question has admitted to being underage via message or post, we may still end up ignoring them despite them self-admitting. This is quite an indefensible enforcement gap"
"Audit U13 escalations that lead to immediate disables: We probably shouldn't be escalating to immediate disables - we have a policy to always allow users to appeal for 30 days. This is a poor user experience and given we lack visibility into the precision of our enforcement (as we lack a ground truth) is likely too extreme an approach"
"Understand why such a high proportion of reported accounts are reported multiple times by the same reporters: 16% of accounts reported for being underage are reported multiple times by the same reporter. Understand why such a high proportion of reported accounts are reported multiple time by different reporters: 8% of accounts reported for being underage are reported multiple times by different reporters. We are less likely to checkpoint these accounts than those reported by only a single reporter. We don't understand why this is. In theory, multiple people reporting an account for being underage should be a strong signal the account is underage. In reality, it is the opposite."
"Deprecate dormant account policy: The dormant account policy automation (auto-checkpoint reported accounts which have been inactive for over 30 days) was only ever shipped as an emergency solution to a large backlog - it is not defensible long term as there's no known correlation between account dormancy and underage users, nor is there any correlation expected."
"Invest in further automations for enforcement jobs: We have flat ops capacity this half. Finding a way to reduce our ops demand with enforcement job automations frees up capacity that we may be able to use for iniatives to drive down underage prevalence"
"Develop 'Adult Age Misrepresentation' VT & Enforcement: We have already established Adult Age Misrepresentation (Adults pretending to be teens) as a new violation type. We still need to development enforcement for this violation. AAM is a serious violation from child safety perspective as evident from linked case study, we currently do not enforce proactivly on age lying."
"Fix EditDOB AutoClassifiers: We run classifiers to determine if we should permit a user's date of birth change. The use of this classifiers for this purpose is broken and means we aren't gating date of birth changes based on our adult classifiers or inappropriate-interactions with children classifier."
"Enable captcha protection on FB reporting contact forms: At the moment our underage reporting contact forms on FB are not as well protected against scripted attacks as they could be. We've previously had a SEV because of scripted attacks on our IG reporting form, which is now captcha protected."
Santhosh Vinod: "We're looking for surface integrity representation for AYMF to reduce preference amplification."
Dennis ████: "what is preference amplification?"
Santhosh Vinod: "Recommender systems can 'rabbit-hole' users into a certain type of content. In the world of integrity, this could specifically lead to recommending unsafe content that might lead to high engagement, however, could result in negative experiences…
We can define Preference amplification from multiple angles:
1. Existing users slowly drifting into policy violating content after engaging with borderline content
2. New users who join Instagram are sucked into an echo chamber due to a cold start problem
3. Regulators creating fake insta accounts to emulate teens to showcase Instagram's inactions in integrity enforcement"
Paniz ████: "I'm writing to enlist some urgent support from IG to mitigate legal and legitimacy risks we're facing in the Adult Sexual Exploitation, Child Safety and Terrorism space.
Context: Adult Sexual Exploitation (StopNCII.org), Child Safety (NCMEC) and Terrorism (GIFCT) space are all problem areas which participate in cross-industry hash-sharing collaborations.
All three arrangements involve Meta receiving hashes from an external body (e.g. NCMEC), which we can use internally to match and remove violating content.
We use PDQ as the hashing algorithm for photos and MD5 for videos.
The problem: All three collaborations were launched with the assumption that MD5, the form of hash used for videos, can be matched on IG. We realised this wasn't the case only in ~January 2022, despite some of these collaborations having been running for years. While the root cause is a feature gap, rather than broken systems, we raised S258090 due to the legal and legitimacy risks involved in falling so short of the public expectations we have set."
"Legitimacy: All three collaborations for Child Safety, Terrorism and Adult Sexual Exploitation are very high-profile given they relate to industry working together to stop high severity harms. Our legitimacy wins from participating in these efforts could easily become a legitimacy loss due to the scale of this feature gap."
Karina Newton: "can someone from CI with knowledge ping me off thread with legal counsel on how this gap was not previously known or tested?"
"But to date, we have limited data on how to speak to parents about our efforts in the well-being and safety space"
"Over the past few years, fewer Teen and Non-Teen respondents are saying that Instagram… is an app that cares about its users (CAU)... is a place to get support and encouragement... is a place where they are comfortable being themselves"
"1. Teens are receiving many unwanted DM requests (DMs from unconnected people), leading to
integrity risk and slowing down new teen acquisition
2. Unwanted DM requests to teens are 'nearly all from people with no mutual followers.
3. We should default teens into a state that prevents others from sending them DM requests if they have no mutual followers."
"DM's are the most problematic interaction vector for teenagers on Instagram, including bullying, harassment, and more severe violations."
"Teens are 2x likely than non-teens to receive an unwanted DM request"
"50% of DM requests to actual teens are from non-teens"
"70% of DM requests to teens are unwanted"
"95% of Unwanted DM Requests to Teens are From People with 0 Mutual Followers"
████: "CWB [Central Well-Being] is advocating that if we want to pursue NAC [Negative Appearance Comparison] measurement, it should go to the VP level"
Justin Cheng: "1. Strategy aside, do they have feedback on the NAC survey questions? 2. Why do they think NAC is part of PU [Problematic Use] (or perceive it as unimportant?"
Justin Cheng: "[S]ounds like next steps is to meet with ████/Pratiti to get buy-in (again) for NAC?
Key notes:- Differences in perception of whether FB/IG 'causes' NAC"
████: "One other note was that they decided not to prioritize 'social comparison', of which 'appearance comparison' and 'negative appearance comparison' are subsets. I think they were generally conflating NAC with social comparison… We went through an evolution on our team to focus in on NAC because we agree that 'social comparison' is not necessarily good or bad. However, external research consistently points to appearance comparison as a key step towards internalization of the 'thin ideal' and other beauty ideals which often lead to body image issues and for some eating disorders/ suicide/ self injury. The prevalence of all of which are historically high and growing-particularly among teens…The fact that these appearance comparisons are made much more frequently on IG vs competitors (including FB) has been shown to be due to the visual nature of the app."
"User impact: Teens receiving DM requests from unconnected adults, breaking public commitment that we made. 2 specific potential issues being investigated:
"Account without predicted age (but with stated age) potentially are not included in the logic to block DM requests from adults to minors. Expected by product team but potentially breaks public commitment. Accounts with predicted age not being blocked from sending/receiving DM requests. Not expected by product team, bug if occurring."
"50% of message requests to predicted teen dau are from predicted non-teens"
"Meta's reporting to the CyberTipline has taken the appearance of a compliance exercise, rather than a crucial child protection mission to ensure children are safe from exploitation on its platforms."
"Earlier this month, Meta advised that over 600,000 otherwise valid reports submitted to the CyberTipline over the past 3 months were missing crucial information as a result of a Meta 'bug' and needed to be resubmitted."
"We estimate that two full-time NCMEC analysts are devoted almost exclusively to correcting Meta's reporting errors."
"NCMEC is gravely concerned about the impact on child safety and the potential scope and magnitude of these reporting errors, as well as Meta's apparent inability to put an end to the continual mistakes."
Ashley Yuki: "Hey John, We want to give you a heads up on an upcoming regulatory driven launch of blocking age collection on IG in order to get your feedback ahead of finalizing the launch decision with Adam and the other IG Leads. This blocking experience would result in a top line DAU regression of -0.48%, with nss revenue impact, but likely negative.
For context, IG has started collecting age in registration globally since GDPR. However, we still have ~16% of DAU who are missing stated age because they signed up before that. With the upcoming regulatory driven launches of Basic Ads and WAIST (Why Am I Seeing This) in Ads, we need 100% coverage of stated age, in order to ensure Teens are put in the required experience and that we're able to be transparent about Ads targeting.
Over the past 6+ months the team has been working on collecting age via non-blocking tactics (e.g. dismissible QPs/prompts) and optimizing the blocking experience to minimize friction for users."
John Hegeman: "Thanks for sharing. For reference: how does TikTok deal with this issue? As a general principle I'm trying to make sure we scrutinize places where we're choosing to be substantially more conservative than the relevant competitive market."
████: "Instagram is a bit of a unique situation. Until a couple of years ago, IG did not ask for a birthdate during new user account creation, except for EU under GDPR. Therefore, IG has close to 16% of users without stated age.
We are trying to close this gap in stated age in order to comply with the upcoming Ads transparency regulations."
"We have data to show that Teens have higher prevalences of violating content than Non-Teens (1.4x-1.8x SWOP), driven by Bullying (1.9x-3.1x) and SSI (1.8x-3.8x) and Cold Start users have higher prevalence of violating content (1.3x-2.8x SWOP) driven by Nudity (1.2x - 4.9x) and Sexual solicitation (1.5x-6.1x) primarily driven by Connected Feed and Profile visits."
Sources include internal documents, public filings and press reports. “Alleged by plaintiffs” denotes the plaintiffs’ characterization, not Meta’s own words. Source labels identify provenance, not a finding of fact.
Meta & Children's Online Safety