Instagram Youth Well-being Assurance Audit - Phase 1 Audit Report

Raw text

Transcribed text as recorded, including headers and formatting fragments.

Instagram Youth Well-being Assurance Audit - Phase 1 Audit Report Company confidential - for intended recipients only Martin Anna Kalyan Yanan Pratima December, 2022 Assurance Audit provides an objective and independent evaluation of the design and operating effectiveness of internal controls. CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 PLTF EX PLTF EX-01518 Meta META3047MDL-050-00352025 METANMAG-007-00351830 PLTF EX-01518_0001 TABLE OF CONTENTS 1. Executive Summary 2. Detailed Results 3. Appendix Communication Channels Audit Report Rating Guidelines Observation Remediation Guidelines Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 2 META3047MDL-050-00352026 METANMAG-007-00351831 PLTF EX-01518_0002 EXECUTIVE SUMMARY Objective: Instagram (IG) leadership has publicly committed to raising the industry standards for helping protect youth well-being and safety online. As part of this work, in 2021 and 2022, IG announced a stricter approach to cater recommendations and implement additional processes for youth. The implementations focused on further reducing the teens' exposure to potential violating or sensitive content or accounts in recommendation surfaces (e.g., Reels, Explore), which extends the basic protection provided by the Cross-Meta Integrity (XI) enforcement that takes down violating content and accounts. Internal Audit (IA) assessed the design and operating effectiveness of the key controls for the IG processes (based on internal guidelines) implemented to further reduce youths' exposure to potential violating or sensitive content and accounts. Scope: IA independently assessed the key controls of IG Youth Well-being in the following 3 areas: Age-based Gating, Filtering & Demotion: Reduce the visibility of potential violating content on connected and unconnected surfaces, e.g., graphic violence, and restricted goods & services. Limiting Interactions: Protect teens from negative interactions with potential violators, including the disabled 'follow' action. Sensitive Content Control: Limit exposure of users to sensitive content and accounts in recommendation surfaces, e.g., posts that may be sexually suggestive. Results Summary: Significant Improvements Needed* While the potential violating or sensitive content and accounts recommended were notably reduced after the above processes were implemented and the age- based content gating framework was established, based on the testing completed as of 11/30/2022, IA identified 3 High risk and 14 Medium risk observations during the audit. Based on the overall effectiveness of controls for in-scope areas, an "Orange" rating is assigned to this report. The results and observation ratings in this report are based on the design and operating effectiveness of internal controls only. Key observation themes are summarized below: Inconsistent filtering on potential Inappropriate Interaction with Children (IIC) accounts and content. Incomplete coverage of recommendation filtering on Minor-Restricted Regulated Goods and Services (RGS) content (e.g., diet products) for teens. Lack of documented rationale for exemption of verified accounts and Suicide & Self Injury (SSI) signal retention period. Missing comprehensive monitoring and alert of filtering volume, leakage, and machine learning performance. * This rating is assigned in accordance with the IA's report rating guidelines in Slide 18 Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 3 META3047MDL-050-00352027 METANMAG-007-00351832 PLTF EX-01518_0003 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Limiting Interactions Observation/Risk Description Rating Inconsistent filtering of LIC account recommendation HIGH The IG Well-being Teen Safety team does not always limit teen accounts from being recommended to the IIC potential violators* and vice versa. As a result, teens' accounts may be recommended to potential violators and vice versa via account recommendation surfaces. For example, on a sampled day tested, on the Accounts You May Follow (AYMF) surface, IA identified: 1) 7.2% (~1.4M out of ~19M) of potential violator accounts in AYMF were recommended to at least one teen. 2) 4.0% (~3.4M out of ~85M) of AYMF recommendations to potential violators were recommending teen accounts. * This control applies to IIC Medium potential violators, defined by the product team as users above 18 without verified tag, having 1 or more Child Sexual Exploitation (CSE) violation in past 90 days or a Community Integrity (CI) classifier for IIC score score greater than or equal to 0.3. Management Action Plan 1. Perform validation testing upon MaCSA migration and investigate any recommendation leakage of IIC potential violator accounts to teen accounts and vice versa. 2. Investigate any potential delays in the propagation of predicted age information from Core Dimensions and mitigate discrepancy, if needed. 3. Implement remediation to further limit exposure between teens and potential violators through AYMF recommendations. Due Date Owner 1&2. 2/28/2023 3. 3/31/2023† Brian (Software Engineer, IG Well- being Teen Safety) Soukaina (Software Engineer, IG Well- being Teen Safety) † Due to complexity of identifying the root cause of the leakage and the migration from the original IIC classifier to the new MaCSA encryption resilient detection method, the IGWB Teen Safety team has requested a 30-day extension to IA's standard 60-day remediation guideline for High risk observations. Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 4 META3047MDL-050-00352028 METANMAG-007-00351833 PLTF EX-01518_0004 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Inconsistent follow list hiding Rating Limiting Interactions HIGH The IG Well-being Teen Safety team does not always hide teen accounts from IIC potential violators* and vice versa on the "followers and following" surface. As a result, teen accounts may be displayed to potential violators and vice versa via the follow lists. Management Action Plan 1. Perform validation testing upon MaCSA migration and investigate teen accessibility to discover IIC potential violators via follower and following lists of other accounts and vice versa. Due Date 1&2. 2/28/2023 3. 3/31/2023† Owner Brian (Software Engineer, IG Well-being Teen Safety) Soukaina (Software Engineer, IG Well-being Teen Safety) For example, on 2022-11-25, IA observed that out of ~2.5M IIC potential violators who viewed the follow list surface, 37% (924K) were shown at least one unconnected teen account. * This control applies to IIC Medium potential violators 2. Investigate any potential delays in the propagation of predicted age information from Core Dimensions and mitigate discrepancy, if needed. 3. Implement remediation to further limit teens' exposure to IIC potential violator accounts via follow list and vice versa. † Due to complexity of identifying the root cause of the leakage and the migration from the original IIC classifier to the new MaCSA encryption resilient detection method, the IGWB Teen Safety team has requested a 30-day extension to IA's standard 60-day remediation guideline for High risk observations. Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 5 ΜΕΤΑ3047MDL-050-00352029 METANMAG-007-00351834 PLTF EX-01518_0005 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Rating Management Action Plan Due Date Owner Complete QE testing and re- launch classifiers for age- based filtering on Minor- Restricted RGS content on 12/16/2022 Alexander (Software Engineer, IG Pending Well-being Teen Safety) Reels, Explore and Feed Validation Recs. Age-based Gating, Incomplete coverage of recommendation filtering on HIGH Minor-Restricted (RGS) content* Filtering and Demotion The IG Well-being Teen Safety team has not completely incorporated Minor-Restricted RGS signals into age- based content recommendation filtering enforcement. As a result, age inappropriate content (e.g., alcohol) may get recommended to teens. * Minor-restricted RGS content includes diet products, alcohol and tobacco. Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 6 META3047MDL-050-00352030 METANMAG-007-00351835 PLTF EX-01518_0006 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Limiting Interactions Observation/Risk Description Rating MEDIUM Management Action Plan Due Date Owner Inconsistent effectiveness of IIC content recommendation filtering The IG Well-being Teen Safety team does not always limit recommendations of teen created content to the IIC potential violators* and vice versa. As a result, teens' content may be recommended to IIC potential violators via content recommendation surfaces and vice versa. For a sampled week tested: 1) 2.2% (~27K out of ~1.2M) of IIC potential violator created content on the Explore surface was recommended to at least one teen, accumulating ~466K teen impressions. 2) 1.6% (~83K out of ~5.2M) of contents recommended to IIC potential violators on the Explore surface were created by teens, accumulating ~200K potential violator impressions. * This control applies to IIC Medium potential violators 1. Perform validation testing upon Malicious Child Safety Actor (MaCSA) model migration and investigate any leakage of IIC potential violator created content to teens on the recommendation surfaces, and vice versa. 2. Implement remediation to further limit teens' exposure to IIC potential violator created content, and vice versa. 1. 2/28/2023 2. 3/31/2023 Brian (Software Engineer, IG Well-being Teen Safety) Soukaina (Software Engineer, IG Well-being Teen Safety) Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 7 META3047MDL-050-00352031 METANMAG-007-00351836 PLTF EX-01518_0007 Management Action Plan Due Date Owner 3/31/2023 Brian DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Limiting Interactions Undocumented exemption of accounts The IG Well-being Teen Safety team exempted verified accounts from being classified as IIC potential violators without documented assessment of the risk or rationale. As a result, potential violating verified accounts may not be classified as IIC potential violators. Rating MEDIUM 1. Reassess the blanket exemption on verified accounts and document the decision and rationale. 2. Implement MaCSA model for potential violator identification based on the reassessment decision. (Software Engineer, IG Well-being Teen Safety) Soukaina (Software Engineer, IG Well-being Teen Safety) Swithin (Product Policy Manager, IG Product Policy) Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 8 META3047MDL-050-00352032 METANMAG-007-00351837 PLTF EX-01518_0008 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description 88888888888888888888888888888888888888888888888888888888888888888888 Sensitive Content Control (SCC 2.0) PRIV Rating MEDIUM Management Action Plan Due Date Owner 1. 3/15/2023 2.4/30/2023 1. Saverio (Associate General Counsel, Data 1. The Legal team will document the data retention requirement specifically for SSI signal. 2. The XI team will implement the data retention period on SSI signals based on updated Legal guidance. Protection) Victoria (Lead Product Counsel, Product (Central Platform) - Legal) Sonny (Associate General Counsel, Privacy Counseling, Integrity) 2. Boris (Software Engineer, X-Meta Integrity) Shiou (Research Scientist, X-Meta Integrity) Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 9 META3047MDL-050-00352033 METANMAG-007-00351838 PLTF EX-01518_0009 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Sensitive Content Inconsistent non-recommendable content Control (SCC 2.0) flagging The IG Relevance Integrity team does not always flag non-recommendable media when the classifier scores exceed predetermined thresholds. As a result, users may be exposed to sensitive content via content recommendation surfaces. For a sampled day tested, IA identified 0.2% of content with nudity classifier scores exceeding the integrity threshold that was not flagged as non-recommendable therefore, not enforced. Rating MEDIUM Management Action Plan Due Date Owner Lucas 1. Investigate and remediate 3/31/2023 the leakage of content with nudity exceeding the predetermined non- recommendable threshold. 2. Set up monitoring and alert on the volume of content exceeding the non- recommendable thresholds. (Software Engineer, IG Relevance Integrity) Bryan (Software Engineer, XI) Sensitive Content Control (SCC 2.0) Lack of monitoring on content filtering leakage MEDIUM Set up monitoring and alert on the non-recommendable content prevalence across all SCC settings. 3/31/2023 The IG Relevance Integrity team does not monitor the leakage level of non- recommendable content getting recommended to users. As a result, the team may not promptly identify issues if an excessive number of sensitive content is displayed on recommendation surfaces. Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL Lucas (Software Engineer, IG Relevance Integrity) META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 10 ΜΕΤΑ3047MDL-050-00352034 METANMAG-007-00351839 PLTF EX-01518_0010 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Sensitive Content Lack of monitoring on account filtering leakage Control (SCC 2.0) The IG Well-being Connections Safety team does not monitor the leakage level of non-recommendable accounts getting recommended on the AYMF surface. As a result, the team may not promptly identify issues if an excessive number of sensitive accounts are displayed on AYME. Rating Management Action Plan Due Date Owner MEDIUM Set up automated alerts to monitor filtering leakage and maintain an on-call runbook. 3/31/2023 Boyi (Software Engineer, IG Well-being Connections Safety) 8888888888888888888888 Sensitive Content Missing alert on non-recommendable account Control (SCC 2.0) prevalence The IG Well-being Connections Safety team has not set up automated alerts to monitor non-recommendable account prevalence on the AYMF surface. As a result, if the prevalence of potential violating accounts on the surface increases, the issue may not be identified and resolved promptly. MEDIUM 1. Investigate discrepancies in 3/31/2023 impressions count for users with the 'Allow' setting on the 'IG AYMF non-rec prevalence' dashboard. 2. Set up automated alerts to monitor non-recommendable account prevalence on AYMF. 3. Develop and maintain an on-call runbook to address and remediate alerts. Boyi (Software Engineer, IG Well-being Connections Safety) Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 11 META3047MDL-050-00352035 METANMAG-007-00351840 PLTF EX-01518_0011 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Sensitive Content Lack of Logging of Content Demotion Control (SCC 2.0) The IG Well-being Enforcements & Classifications Platform team does not log the real-time demotion score* and dynamic threshold for non-recommendable content. As a result, the demotions applied cannot be measured or compared against the original expectation. * demotion score indicates the expected demotion strength Rating Management Action Plan Due Date Owner MEDIUM 1. Migrate late stage demotion onto Soft Action Kaleidoscopic Enforcement (SAKE) platform. 1. 4/30/2023 2.5/31/2023† Adam (Software Engineering Manager, IG Well-being Enforcements & Classifications Platform) 2. Implement logging of the real-time demotion score and dynamic threshold. Sensitive Content Inconsistent content filtering and demotion Control (SCC 2.0) monitoring The IG Well-being Enforcements & Classifications Platform team does not always investigate and address filtering and demotion performance alerts promptly. As a result, content filtering/demotion issues may not be identified or resolved promptly. MEDIUM 1. Implement and maintain 3/31/2023 an on-call shift summary tracker to: log alerts with assigned oncall information, and document relevant details on the issue/ resolution to provide context for future alerts 2. Establish SLA to close alerts promptly. Adam (Software Engineering Manager, IG Well-being Enforcements & Classifications Platform) † Due to complexity of migration to SAKE, collaborative development platform for integrity enforcements across Instagram, the IGWB Enforcements & Classification Platform team has requested a 30-day extension to IA's standard 120-day remediation guideline for Medium risk observations. Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 12 ΜΕΤΑ3047MDL-050-00352036 METANMAG-007-00351841 PLTF EX-01518_0012 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Rating Management Action Plan Due Date Owner Sensitive Content Lack of input monitoring of P(Severe) v2 model Control (SCC 2.0) MEDIUM The IG Well-being Teen Safety team does not monitor the input signal coverage of the P(Severe) v2 model* to assess its reasonableness. As a result, low coverage of specific signals (e.g., profile picture signal) may go undetected, leading to deteriorated model output. Set up monitoring and alert on 3/31/2023 ML input signals for the P(Severe) v2 model in Pranav (Software Engineer, production. Machine Learning, IG Well-being Teen Safety) * P(Severe) v2 model is the machine learning model that uses profile element approach to identify violating and non-recommendable profiles. 888888888 Sensitive Content Control (SCC 2.0) Lack of performance monitoring of P(Severe) v2 model MEDIUM The IG Well-being Teen Safety team does not monitor the performance (e.g., precision, recall) of the P(Severe) v2 model. As a result, deteriorating model performance over time may go undetected. 888888888 1. Set up monitoring and alert 3/31/2023 on performance (Precision/ Recall) of the P(Severe) v2 model. 2. Establish model refresh cadence based on performance monitoring of P(Severe) v2 model. Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 Pranav (Software Engineer, Machine Learning, IG Well-being Teen Safety) 8888 13 META3047MDL-050-00352037 METANMAG-007-00351842 PLTF EX-01518_0013 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Sensitive Content Incomplete logging of account demotion Control (SCC 2.0) The IG Well-being Discovery Safety team does not completely log account demotion on the search surface. As a result, the effectiveness of account demotion cannot be measured or compared against the original expectation. surface Sensitive Content Lack of account demotion monitoring on Search Control (SCC 2.0) The IG Well-being Discovery Safety team does not monitor demotion volume on the Search surface. As a result, account demotion leakage may not be identified and resolved promptly. Rating Management Action Plan Due Date Owner MEDIUM Set up logging of pagination subrequests in the account demotion logging table. 3/31/2023 Sadegh (Research Scientist, IG Well-being Discovery Safety) MEDIUM Set up monitoring and alert for 3/31/2023 the Search surface on: Demotion volume Demotion rate, and Prevalence of non- recommendable accounts on Search across SCC settings. Sadegh (Research Scientist, IG Well-being Discovery Safety) Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 14 META3047MDL-050-00352038 METANMAG-007-00351843 PLTF EX-01518_0014 DETAILED RESULTS AND MANAGEMENT ACTION PLANS Business Area Observation/Risk Description Rating Management Action Plan Due Date Age-based Gating, Lack of monitoring on age-based recommendation MEDIUM Filtering and Demotion filtering The IG Well-being Teen Safety team does not monitor the input signal coverage and volume for age-based filtering of the potential violating Adult Nudity and Sexual Activity (ANSA) & RGS content. As a result, anomalies of filtering may not be detected. 1. Set up monitoring and alert to track filtering volumes of ANSA & RGS content and get notifications on 1. CLOSED 2.2/28/2023 significant changes. 2. Create a dataswarm pipeline to measure input signal coverage and set up monitoring alert on the coverage. Owner Alexander (Software Engineer, IG Well-being Teen Safety) Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 15 META3047MDL-050-00352039 METANMAG-007-00351844 PLTF EX-01518_0015 APPENDIX I - COMMUNICATION CHANNEL (1) Max Eulenstein Morgen { Sayed Oraru Rishi Observation Owner Fiona Andrew Nam Nguyen Chris Cox Adam Mosseri है Roger Charles Sismondo ? Kripa ? David Ashish Brien Soukaina Robin Adem Alexander Yuting Boy Sadenh Shweta Jeff Me Asmeek Nima Ricardo Ben Product Analytics/Data Engineering Jenni Romanek ? Roberto Nino sewa Kaboabe Ethan Sharon Ashish ? ? Erion Neeline Don Mingliang Timothy Anestasia Daniel ? Merzin Megan Usman Lucas Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 16 META3047MDL-050-00352040 METANMAG-007-00351845 PLTF EX-01518_0016 APPENDIX I - COMMUNICATION CHANNEL (2) Nikhil Yvonne Jennifer Newstead Jonathan D. Martin Scott Tucker Sandeep Nazneen Cathlin James Turmer Jeana Faye Yul Mel Bathilde John Rachany Saverio Sonny ↑ Nicole Policy ↓ Nick Clegg Joel Kaplan ↑ Monika Bickert Neil Potts Antigone Davis Andy O'Connell ? Liz David Ginsherg Tucker Bounds ? Lena Pietsch Stephanie ↑ Vaishnavi Ravi Sinha Swithin Observation Owner Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL Mike Victoria ? Chris Sonderby Henry Moniz ? Karen Joseph Michael Nicola oss-Meta integrity (XI) Javier Olivan Naomi Gleit Guy Roser ↑ Joey Simhon Jason Shenghao ? Lion Jesse Gary Baptiste Aswin Gary ? Boris Bryan Shiou META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 17 META3047MDL-050-00352041 METANMAG-007-00351846 PLTF EX-01518_0017 APPENDIX II- AUDIT REPORT RATING GUIDELINES Rating Rating Criteria Limited "Low risk" observations: Effective Processes and controls provide reasonable assurance that risks are managed and objectives will be met. Limited "Medium risk" observations or a number of "Low risk" observations: Some Improvements Needed Processes and controls require improved effectiveness and efficiency to provide reasonable assurance that risks are managed and objectives will be met. Limited "High risk" observations or a number of "Medium risk" observations: Significant Improvements Needed Processes and controls include gaps that require significant improvements to provide reasonable assurance that risks are managed and objectives will be met. Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL Several "High risk" observations: Processes and controls include critical gaps that require immediate improvement to provide reasonable assurance that risks are managed and objectives will be met. META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 18 META3047MDL-050-00352042 METANMAG-007-00351847 PLTF EX-01518_0018 OBSERVATION REMEDIATION GUIDELINES Rating High Risk Rating Definition Process and control gaps that may result in (i) significant deficiency or material weakness in internal controls, (ii) severe decrease in operational effectiveness/efficiency, (iii) violations in policy, laws or regulations, or (iv) damage to Meta's reputation leading to significant business impact. AROC Remediation Expectations Communication Resolve issue within 60 days Yes Medium Risk Process and control gaps that may result in (i) deficiency in internal controls, (ii) significant decrease in operational effectiveness/ efficiency, (iii) non-compliance with policy, laws or regulations, or (iv) damage to Meta's reputation leading to moderate business impact. Target remediation within Yes 120 days Low Risk Process and control gaps that may result in (i) minor non- compliance with policy, laws or regulations, (ii) deficiency in internal Target remediation within controls, or (iii) decrease in operational effectiveness/efficiency. 180 days Company confidential - for intended recipients only CONFIDENTIAL CONFIDENTIAL META3047MDL_REPROD_VOL021 METANMAG_REPROD_VOL016 No 19 META3047MDL-050-00352043 METANMAG-007-00351848 PLTF EX-01518_0019