[IIC on IG] PRI Analysis Quip v1

Raw text

Transcribed text as recorded, including headers and formatting fragments.

[IIC on IG] PRI Analysis Quip v1 Executive Summary The investigation conducted several tests looking at inappropriate interactions with minors on Instagram. We hope the findings can be useful for CO, CI, and Instagram teams who continue to work to prepare for encryption and who are prioritizing the safety of minors on the platform. Tests were conducted looking at the current state of IIC on Instagram and learnings are intended to be used to inform next steps in preparation for end to end encryption. IIC enforcement currently focuses heavily on detecting real world harm (such as grooming, incest, bestiality, and position of trust) and online harm (CEl and sexual conversations). These abuses largely occur in messenger. The policy, however, describes IIC more broadly and could also include unsolicited interactions in comments as well as DMs. Furthermore, it is possible that unsolicited contact isn't always sexual in nature and the current approach and actioning of other abuse does not take reporter age into consideration. A more holistic approach to child safety may put us in a better position to protect minors from both unsolicited sexualized interactions from adults as well as unsolicited non-sexual interactions in comments and messages protect minors. [Updated 6/18] Cause Analysis Summary Problem 1 IG recommended a minor through top suggested to an account engaged in groomer-esque behavior. 2 3 3 accounts engaged in IIC remained live on the platform even after FB received signal that they were engaging in child exploitation. 4 Recommendation Root Cause The recommendation algorithms worked as intended. 1. Product Establish a whitelist for recommendations on IG. There is not currently a whitelist to determine which accounts can and cannot be recommended in top suggested. 3.a.Protocol This was the most relevant option for the rep. 3.a.Protocol The ops guidelines for this queue call for CEl review and Add escalation path for reps at so to escalate jobs for IIC. Update ops guidelines and add more context to include age and photos of reporter and responsibl CONFIDENTIAL CONFIDENTIAL PLTF EX PLTF EX-01730 ΜΕΤΑ3047MDL-111-00149153 METANMAG-021-00151987 PLTF EX-01730_0001 there was no CEI. 3.a.Protocol These accounts did not meet 5 Minors reported adults interacting inappropriately in the 30% threshold. 3.a. IGDM and we left the accounts up. 6 Minors have a limited ability to prevent receiving violating unsolicited messages from adults. 7 Adults interacted with minors inappropriately in comments. Protocol The functionality does not exist. 1. Product We don't explicitly call out commenting in our IIC policy. 2. Policy so that there is sufficient informat to action for IIC. Update account disable threshold IG and explore opportunities to incorporate age in determining severity of actions. Consider adding functionality to tu off requested inbox. Update IIC policy to include comments as an example and cla behaviors that are and are not IIC 8 9 We have not allocated Consider proactive and reactive resources to preventing IIC in opportunities to address IIC in comments. 3.a. Protocol comments. The default private settings for minors do not limit who can comment. 1. Product Update default privacy settings for minors to be more secure. [Updated 6/18] Discussion for PRI CIPCO 1. Do we want to be taking a more holistic approach (surface and behavior) to combating IIC on the platform? 2. If so, how can we ensure that age is taken into account throughout our prevention and detection efforts? a. This includes: i. Default Privacy Settings ii. Recommendations iii. User Messaging iv. Report Routing v. SRT Context vi. Severity of Actions 3. How can we most effectively incorporating different IIC violator personas into our prevention and detection efforts? Background Inappropriate Interactions with Children is a complicated abuse area that encapsulates numerous violation types ranging from creepy to criminal (see policy definition in the appendix). The behaviors of victims, abusers, and the interactions between the parties CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149154 METANMAG-021-00151988 PLTF EX-01730_0002 can vary according to the particular relationship and violation type. Currently, in order to identify something as IIC we have to know the relative age of the two parties (adult and minor) and the two must be engaged in sexualized conversation. This has been historically challenging to enforce on IG both because we do not collect age of users and the detection of sexualized conversation usually occurs in messages which receive limited review and will no longer be accessible going forward (unless user reported). In order to continue to protect younger users it is imperative that we continue to improve proactive detection, upstream prevention, and up-sell and correctly action user reports for this abuse area. Due to existing work in this area across numerous CIPCO teams, this investigation focused largely on discoverability or upstream prevention and reactive reporting flows or user reports. Testing Discoverability • Hypothesis: We may be facilitating possible groomers finding young people through following, hashtags, explore, and search. Evidence: • Recommending groomers follow minors Tests: Overall, 7% of IG recommendations are related to cases where we recommended teens to non-teens. This is compared to 26% of people we recommended to Groomers were Teens: which is way above the IG teen-non_teen recommendations threshold. This could be due to different factors, as indicated in the recommendation sources below, one of which could be the normal followers graph, and other include, similar hashtag interest, facebook friends etc. 22% of the teens recommended to Groomers, received a follow request from Groomers when the recommendation was shown to them. In other words, groomers clicked on the follow button 22% of the time after we recommended a teen to them. Ο Test 1: Make test account, follow a bunch of grooming-type hashtags, see if platform recommends children as people to follow. Ο i.e. sexyteens, hotteen, sugardaddy, gayboysex, boy_and_boy, sexyteengirls Test 2: Make test account, follow a bunch of hashtags, see if platform recommends other creepy hashtags Ο Test 3: Make test account, follow a bunch of creepy hashtags, see what gets recommended in search. CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149155 METANMAG-021-00151989 PLTF EX-01730_0003 CONFIDENTIAL CONFIDENTIAL Ο Test 4: Make test account, follow several teen/grooming accounts, see if platform recommends other teens to follow. Test 5: Make test account, follow several teen/grooming accounts, see if platform recommends questionable hashtags to follow Ο Test 6: Make test account, follow several teen/grooming accounts, see what gets recommended in search Results: Ο Ο We are facilitating adults finding minors and minor-focused content through recommendations and explore. Tests 1-3 Hashtags on their own did not result in much content being served to us. In other words, following hashtags did not significantly alter search, explore, our feed, or produce other similar hashtag recommendations. Some hashtags like #sugardaddy and #gayteen, however, would result in teen photos appearing in stories. While a significant amount of minor-focused content may not be recommended to adults if they follow hashtags, a dedicated user would be able to surface photos of minors with very little effort and comment or send a DM to those accounts if they choose. The primary way we were able to surface content was by using hashtags and seeking out photos of minors that were sexualized in some way. We then used that account to fanout and look at other followers to see what hashtags they used and what accounts they followed. Users that follow minors' accounts will be recommended similar accounts as well as photos of teens in explore. Ο Tests 4-6 弱 Problems: Following teen modeling accounts, borderline CEl accounts, and minors resulted in our being served a lot of similar content in explore. Numerous accounts were dedicated to borderline CEI and aggregated teen photos of teens in bikinis, for example. Many of those account photos had a significant number of comments sexualizing the photo even if the photo itself was not violating. Ο Adults are easily able to find minors to inappropriately interact with on IG. Root Cause(s): Ο IG algorithm does not take in to account violations when serving content. Sexualized teen accounts exist on the platform because it is not clear that they violate (age is hard to determine) and we aren't set up to enforce against them at scale. META3047MDL-111-00149156 METANMAG-021-00151990 PLTF EX-01730_0004 0 We don't take age into account when deciding what to recommend to someone. Risk Area: Harmful content Problem 1 IG recommended a minor through top suggested to an account engaged in groomer-esque behavior. 2 Intermediate Cause The account followed and searched for sexualized minor accounts. Root Cause The recommendation algorithms worked as intended. 1. Product There is not currently a whitelist to determine which accounts can and cannot be recommended in top suggested. 3.a.Protocol Rec Estab for recom on IG Reactive Reporting - IGDM • Hypothesis: We may be missing IIC content on IG that is reported through reactive reporting. Evidence: Teens report rate for DMs on IG is at 51% of which 40% is related to IIC cases. Ο 20% of All DM reports are related to IIC cases (where the reporter is a teen and the sender is a non-teen Test: Pull a list of 100 recent IG DMs that have been escalated for child exploitation (there is no option to report for IIC). Go through and manually label them to see if any are IIC. Results: Ο Labeled 50 jobs that were marked for child exploitation, marked those where IIC was taking place 6% Confirmed as IIC Ο 20% Labeled as Possible IIC Problem: Ο We are missing IIC in at-scale review of IGDMs. Ο Minors are subjected to IIC in comments. Root Cause(s): • Reporting process is confusing. Ο Ο Ο Jobs cannot be escalated for IIC currently within SRT. It is difficult for reps to determine age of reporter and responsible. There is not a clear way for minors to report IIC on IGDM jobs. • Age is not a factor in how we triage reporting. Ο There are limited resources currently allocated to prevention efforts. Risk Area: Harmful Content Problem Statement Proximate Cause Intermediate Cause Root Cause CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149157 METANMAG-021-00151991 PLTF EX-01730_0005 1 3 accounts engaged in IIC remained live on the platform even after FB received signal that they were engaging in child exploitation. 2 Problem 1 Minors reported adults interacting inappropriately in IGDM and we left the accounts up. They were escalated for child exploitation. This was the most option for the rep. 3.a.Protocol The ops guidelines queue call for CEI n and there was no CEI. 3.a.Protocol Proximate Cause Intermediate Cause Root Ca We only remove accounts for violating DMs if 30% of their DMs violate. These accounts meet the 30% threshold. 3.a. Problem 1 Minors have a limited Proximate Cause ability to prevent receiving violating unsolicited messages from adults. Intermediate Cause Root Ca The functionalit exist. 1. Produc Reactive Reporting - Comments • Hypothesis: We may be missing IIC content on IG that is reported through reactive reporting. Evidence: In the course of conducting discoverability tests we found numerous incidences of sexualization of teens through comments in addition to observing other types of possible IIC in comments i.e. bullying. Test: Observe comments on teen accounts and determine how and if we are actioning comments for IIC. Results: Ο IIC is happening in comments. In one month there were approximately 8 million comments reported on IT, 7.5% of those comments were a teen reporting a comment from an adult • Nudity, 18% of all comments deleted were a teen reporting a non-teen CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149158 METANMAG-021-00151992 PLTF EX-01730_0006 Problem: Hate speech, 52% of all comments deleted were a teen reporting a non-teen • Harassment/bullying, 55% of all comments deleted were a teen reporting a non-teen • Violence, 36% of all comments deleted were a teen reporting a non-teen Adults interacted with minors inappropriately in comments. Root Cause(s): Ο Ο Ο We don't explicitly call out commenting in our IIC policy which has contributed to not building out protocol and enforcement of IIC in comments. We have assumed that comments can be dealt with through self- remediation. We have been focused on more severe cases that can be referred to law enforcement. Risk Area: Harmful content and reputation Ο As a result of IIC in comments and the ability to DM minors they don't know, Instagram could be seen by many parents, media outlets, and governments as an unsafe platform for minors. • Furthermore, IIC in comments is highly visible and opens us up to further scrutiny and questions about why we are not doing more to protect minors. • The platform continues to receive negative international media coverage for its' handling of inappropriate interactions with minors as well as more serious cases of grooming. Lack of enforcement and prevention may normalize IIC and in particular adult sexualization of minors. • Moreover, it may embolden adults to engage in behavior that they otherwise wouldn't. Ο Sexualization of minors, as a particular form of IIC, in comments may lead to more serious forms of online and offline harm. Problem 1 Adults interacted with minors inappropriately in comments. Proximate Cause We do not have protocols and enforcement to action against IIC in comments. Intermediate Cause There is not widespread clarity that comments can violate for IIC. Root C We don't explicitly commenting in ou Policy 2 3 Privacy settings are often not preventing inappropriate Minors are not using privacy settings to prevent adults they are not We have not alloc to preventing IIC i comments. 3.a. Pa The default private minors do not limi comment. 1. Proc CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149159 METANMAG-021-00151993 PLTF EX-01730_0007 interactions in comments. connected with from commenting on their media PPPEP Breakdown Root Cause 1. Product 2. Policy 3. Protocol a. Process is missing or incorrect b. Documentation is missing or incorrect c. Education was not provided 4. Enforcement a. Problem was detected or user reported i. Report was not looked at ii. We looked at the report but tools were bad iii. We looked at the report but rep made a mistake b. Problem was not detected or user reported i. Conscious decision to have lower precision/recall tradeoff ii. Unintended issue 5. People Disagree % of total 33% 11% 56% CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149160 METANMAG-021-00151994 PLTF EX-01730_0008 Appendix Background Namrata Bostrom is with Christopher Palow. May 13 at 11:14 AM CONFIDENTIAL CONFIDENTIAL SCA Guy Rosen is with Namrata Bostrom and 4 others. May 13 at 11:02 AM SCA META3047MDL-111-00149161 METANMAG-021-00151995 PLTF EX-01730_0009 • 2. Inappropriate interactions with children (IIC) such as: 1. Initiating unsolicited contact with minors (for example, private messages between stranger adults and minors) 2. Soliciting, displaying, sharing, or viewing imagery of nude, sexualized, or sexual activity with minors 3. Arranging real-world sexual encounters or obtaining sexual material from a minor directly. 4. Displaying nudity to minors • Minors are subjected to IIC in comments. Ο Below are screenshots of comments where reporter was a minor and responsible was an adult. Report reasons below include hate speech, bullying/harassment, and nudity. instagram_comment_text "Whipping the black students in my class for refusing to instagram_comment_text pick cotton" "@futuremodel eww yuck you scary uglyy" Text Text instagram_comment_text "Bend ya over and pound tat sweet juicy pussy with my fat thick ten inch cock balls slapping off your ass cheeks deep babe" Text Ο Minors are also sexualized in comments which may also qualify as IIC under the letter of the policy. We are missing reports of grooming and inappropriate interactions with children in DMs at scale. User reporting process CONFIDENTIAL CONFIDENTIAL ۵ Q Report META3047MDL-111-00149162 METANMAG-021-00151996 PLTF EX-01730_0010 CONFIDENTIAL CONFIDENTIAL • SRT Community Ops Key Stats Last 12 hrs QUESTIONS Back Reset 1 Previous Review; 0 Note Responsible ANSWERS Content in Review What action needs to be taken? Ignore @nelsonharner 1 Pests 47 Followers 173 Following Delete 2 Checkpoint 3 Escalate Mark As SCA Reporter @nelsonharner Nelson Harner DECISION DETAILS ACTIONS 1-Label Pass To SME K-Skip Submit [Enter] text_content CLIP 48 Posts 107 Followers 134 Following A @aliceangel522 alice angel 13 years old Taken by: @drawing...lover. Pip by: me SCA Adults are easily able to DM minors that they find on Instagram. ถ Snstagram trmgirls 7:18 SCA Q Θ torydenden Tory Teen Day Light in Bad Dreams- 000x mom ✓Official// 48. Sk Followers 4,312 Following Followtimgirls Follow 用 Message 7:18 Howr Text 7:19 terydenden Send SCA I'm 1 thanks 4 1 2 3 4 5 6 7 8 9 0 AQ♡ • "Sexy teen" accounts are on the platform. @#$ & < * +()/ !? ABC 12 ▼ t META3047MDL-111-00149163 METANMAG-021-00151997 PLTF EX-01730_0011 CONFIDENTIAL CONFIDENTIAL Instagram slavvgirl Q Search Follow 15 posts 15.4k followers Katya BestGirl POSTS TAGGED 81 following SCA 0 META3047MDL-111-00149164 METANMAG-021-00151998 PLTF EX-01730_0012 CONFIDENTIAL CONFIDENTIAL ff Photo joven16citas ... + 8:40 SCA : + 口 META3047MDL-111-00149165 METANMAG-021-00151999 PLTF EX-01730_0013 CONFIDENTIAL CONFIDENTIAL ff39 WOW HOT Photo TEENS wow_hot_teens + SCA Q♡ 1,245 likes View all 32 comments May 7 Q + 11:11 : 口 META3047MDL-111-00149166 METANMAG-021-00152000 PLTF EX-01730_0014 Discoverability Observation Summary - [DRAFT] Personas This table represents a culmination of observation through conducting discoverability testing, note on [ HYPERLINK "https://fb.workplace.com/notes/leah- treitman/understanding-predatory-behavior-upstream/199636287446261/" \h ], and some preliminary data work on groomers. 1 2 A 3 Motivation 4 Focused on age 5 IG use 6 7 8 9 B [Draft] IIC Adult Personas/Scale Interested Not motivated Age agnostic Looking for sexual content in general C Exploratory Somewhat motivated More motivated to seek out photos of minors May DM a few minors they gravitate towards Might use the platform to Largely consumers of post their own photos May use IG regularly for activities not involving minors photos of minors May come across borderline CEI accounts including accounts dedicated to posting 'sexy teens' May follow several to many borderline CEI accounts D Want to interact online More motivated Looking for minors Sexual comments including dmme May send photos May solicit photos May DM many minors May engage in sextortion 1 0 1 1 1 2 1 Search May be heavy 3 consumers of porn May follow/seek out accounts that involve minor CEI? may look May use hashtags to look at photos of minor CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149167 METANMAG-021-00152001 PLTF EX-01730_0015 1 4 1 5 1 Possible 6 Characteristics 1 7 May not know the for links off platform and use IG to find dropbox and Patreon accounts? May or may not know accounts with minors are have insight in to their minors If early 20s or late teens may not see/understand attraction in young people to be harmful (wording?) May be young themselves 17+ May be more likely to present true identity attraction as a bad behavior Situational - may be attracted to minors and are able to follow/consume sexualized content because it is widely available online and on IG. May be aware of social norms May recieve spammy messages from adult porn accounts Might use the platform to post their own photos May seek out particula types of photos/accounts i.e. minors displaying attention seeking behaviors, vulnerable youth, etc May follow borderline CEl accounts May follow CEI accounts 1 8 1 Possible 9 Company Approach 2 0 2 1 Reduce discoverability of content Taking advantage of content, increasing risk and discoverability of content will mitigate likelihood they follow or engage with minors May not exhibit enough obvious behavior to warrant pushing a warning Reduce discoverability Opportunistic, but may not engage in behaviors if percieve higher risk Potential groomer interstitial May be posing as a photographer/legitimate party to get minors to engage/trust them and send photos Opportunistic, but may not engage in behavior if percieve higher risk Potential groomer interstitial CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149168 METANMAG-021-00152002 PLTF EX-01730_0016 2 2 Panopticon may be effective to prevent further searches, violations, or escalating behaviors Panopticon may be effective to prevent further searches and violate 2 3 2 Examples TBC hameedullah4866 nigel.cooke.794 4 2 Minor likelihood Unlikely Likely Conditional 5 to report (cultural dependencies, offline support network, awareness of self, relationship with abuser) For “Want to Interact in Person” or accounts actioned for grooming we found that: • Only 105 groomers used hashtags when posting photos. Also, since 6% of the total photos posted by Groomers contained at least 1 hashtag, it seems some groomers are either posting more photos than others or some groomers use more hashtags than others. Ο This might indicate that hashtags are not a common way to surface grooming behavior. • Also, the median of number of photos posted during the last 3 months bu users is 7. With the highest number of photos created/posted being 578. This might indicate that groomers are not very interactive i.e. posting and interacting with their followers through home. Risk Areas Harmful content Ο IIC as it is defined in the letter of the policy is occurring on the platform on more surfaces than we are currently looking at. Ο Minors can be subjected to IIC when they use IG. Ο Ο Lack of enforcement and prevention may normalize adult sexualization of minors. Moreover, it may embolden adults to engage in behavior that they otherwise wouldn't. In addition, sexualization of minors in comments may lead to more serious forms of online and offline harm. • Reputation As a result of adult sexualization of minors and IIC through comments and the ability to DM minors they don't know, Instagram is seen by many parents, media outlets, and governments as an unsafe platform for minors. CONFIDENTIAL CONFIDENTIAL META3047MDL-111-00149169 METANMAG-021-00152003 PLTF EX-01730_0017 0 The platform continues to receive negative international media coverage for its' handling of inappropriate interactions with minors as well as more serious cases of grooming. Proactive Risk Investigation Specs • Investigation Master Task: T43728516 Incident review primary investigators: Luke Incident review analyst: Yosra PRI presentation date: June 20th • Contributors: Akum CONFIDENTIAL CONFIDENTIAL & Jourdan Chris Namrata Karina Newton, Colin Irem Vainius Bernat Chelsea Swithin Zach Liz Taylor Michelle Gabriela Matt Arielle Nick Nikhil Tyler Aditi Bettina Shauna Antigone Davis, Anna Jeremy Tammy Aditya Aparajitha Christine Bertram Brian Sarah META3047MDL-111-00149170 METANMAG-021-00152004 PLTF EX-01730_0018