Meta & Children's Online SafetyEmail: Integrity Support for IG Live
Raw text
Message From: Sayed Otaru [/O=EXCHANGELABS/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN= Sent: 11/1/2023 10:07:54 PM To: Rishi @meta.com] Subject: Fw: Integrity support for IG Live FYI only From: Sayed Otaru < @meta.com> Sent: Monday, October 9, 2023 11:03 PM To: Max Eulenstein < @meta.com>; Ashley Alexander Tomer Bar @meta.com>; Morgan @meta.com>; Maya Subject: Integrity support for IG Live @meta.com>; @meta.com>; Amy @meta.com> @meta.com>; Emily Dalton Smith @meta.com>; Sridhar @meta.com>; Fiona Updated email subject and distro Hi IG and Sharing/Communities Leads, Across XI, IG Sharing, and IGWB, we have aligned our approach to sustainably close the integrity gaps on IG Live that we reported in August. Here is the summary: (1) XI have committed to mitigating the high SWOP on IG Live, including achieving a -53.8% (12.3% > 5.68%) improvement in SWOP following a recent Adult Nudity & Sexual Activity (ANSA) classifier update. There is still important work ahead to further reduce SWOP including reducing the high Child Sexual Exploitation (CSE) prevalence. We will continue to push for, and track these improvements with XI. (2) IG Sharing will address foundational integrity gaps on IG Live and aim to achieve a 2-DERISKED surface maturity status in 2024. This means building understanding, monitoring and tools to support the management of integrity risks including for more comprehensive feature gating. To achieve this outcome, the IG Sharing team are seeking 2 ENG headcount. IG Sharing would need to tradeoff on Teen Live pDAU and Teen Cap 15 Sessions to support this work without this additional headcount. IGWB would have needed to tradeoff on Teen Safety or Threads Integrity goals to support this work. Without this investment we run a high risk of integrity escalations during moments of crises. (3) IGWB will continue to have overall accountability for Integrity on Instagram, including coordinating with XI and Policy teams. In addition, IGWB will remain responsible for tracking topline IG SWOP and providing the playbook for IG Live to monitor integrity prevalence and escalate. IGWB also plan to expand IG's teen safety protections to IG Live and reduce the risk of teens encountering age- inappropriate content on livestreams. IGWB will fund this work from within our current teams. CONFIDENTIAL CONFIDENTIAL PLTF EX PLTF EX-04322 PLTF EX-04322_0001 ΜΕΤΑ3047MDL-278-00049218 METANMAG-130-00049218 Let us know if you have any feedback or questions, and your guidance on the IG Sharing team's 2 headcount request. You may find the detailed traffic light for the funding request and associated tradeoffs here. PS: As the IG Sharing team completes an ongoing strategy refresh for IG Live in the next few weeks, they will come back to this thread to highlight any material delta in the impact to Integrity. On behalf of IG Sharing and IGWB, Sayed. From: Sayed Otaru < @meta.com> Sent: Tuesday, August 15, 2023 5:40 PM @meta.com>; Emily Dalton Smith @meta.com>; Amal To: Morgan @meta.com>; Rocky @meta.com>; Amy Cc: Max Eulenstein < @meta.com> @meta.com>; Shreekar Subject: Re: [Heads Up] Bosnia murder live-streamed on IG @meta.com>; Sridhar @meta.com>; Tomer Bar @meta.com> I'll work with Sridhar and Amal (while Maya is on leave) to frame a tradeoff that will span unshipping IG Live to sufficiently mitigating the high integrity risks on IG Live at the expense of other work on IG. To Morgan's point, the surface maturity level of IG Live is currently at 1-BASIC (having basic, defensible integrity solutions). The tradeoff exercise will be partly informed by the work Sharing/Live would need to do to get to 2-DERISKED (understanding, monitoring, and managing primary risks) or 3-OPERATIONAL (strategically managing risks and tradeoffs at scale). Sayed. From: Morgan @meta.com> www Sent: Monday, August 14, 2023 8:37 AM To: Emily Dalton Smith @meta.com>; Sayed Otaru < @meta.com>; Amal @meta.com> Cc: Max Eulenstein < @meta.com> @meta.com>; Sridhar @meta.com>; Rocky @meta.com>; Tomer Bar @meta.com>; Amy Subject: Re: [Heads Up] Bosnia murder live-streamed on IG Thanks Sayed - it would be great to understand the work to move from BASIC-1 to DERISKED-2 for Live surface maturity and whether we think DERISKED would be sufficient. From: Emily Dalton Smith < @meta.com> Date: Sunday, August 13, 2023 at 7:05 PM To: Sayed Otaru < @meta.com>, Sridhar CONFIDENTIAL CONFIDENTIAL @meta.com>, Rocky @meta.com>, Tomer Bar @meta.com>, Amy @meta.com>, Amal @meta.com> PLTF EX-04322_0002 ΜΕΤΑ3047MDL-278-00049219 METANMAG-130-00049219 Cc: Max Eulenstein < @meta.com>, Morgan @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG That sounds right to me. Is this something you can drive or do you need us to pull in the Sharing team? From: Sayed Otaru < @meta.com> Date: Friday, August 11, 2023 at 9:05 PM To: Emily Dalton Smith < @meta.com>, Sridhar Amal Cc: Max Eulenstein < @meta.com>, Morgan @meta.com>, Rocky @meta.com>, Tomer Bar @meta.com>, Amy @meta.com>, @meta.com> @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG IG team only, CC Max/Morgan, and BCC Amit for transient visibility as he and I very briefly chatted on this earlier today. Hi Emily and Tomer, There is some additional context you should have on Integrity for Live. Over the past three days on IGWB, we've been evaluating the integrity risks on IG Live partly stemming from another escalation on IG Live co-hosting risks (streaming porn and allegedly CSAM.). The TL;DR is that from an integrity POV, we are collectively (XI and IG) underinvested in IG Live. It is our riskiest integrity surface with 13% SWOP where overall IG SWOP is < 2%. Contextualizing this further, IG Reels has a violating nudity prevalence of 6 bps while IG Live is at 7% i.e. 100x times more than Reels. You can read more in this post from our partners on XI, and we filed a SEV here S356462. @Rocky also wrote a very thoughtful note after the unfortunate mass murder in Kentucky streamed on IG Live earlier in April this year. It touches on some of the same themes and also on the efficacy of our RGS polices in the context of the risk on Live. There have been some recent strides to improve integrity like the FPS classifier launch, and the ongoing H2 2023 work by XI to improve the ANSA classifiers for Live (on both IG and FB). As I noted in my email below, I think we should review our priorities and tradeoffs across IG Sharing, IGWB, and XI. For clarity, I don't think the effort to shore up integrity on IG Live should only be classifier based, there can be integrity product innovation on surface as well. IG Live is not a priority for IGWB, and IG Sharing is not prioritized for surface integrity level work. I believe these increasing risks and escalations warrant us to evaluate all our options to land in a better place, because the status quo is not good enough. What do you think? I would like to hear others' thoughts as well. Sayed. From: Emily Dalton Smith <@meta.com> Sent: Friday, August 11, 2023 5:43 PM CONFIDENTIAL CONFIDENTIAL PLTF EX-04322_0003 ΜΕΤΑ3047MDL-278-00049220 METANMAG-130-00049220 To: Ashley @meta.com>; Sridhar @meta.com>; Sayed Otaru @meta.com>; Amit Bhattacharyya < @meta.com>; Arcadiy @meta.com>; Kate Gotimer @meta.com>; Gal @meta.com>; Piyush @meta.com>; Arjun @meta.com>; Rocky @meta.com>; Amal @meta.com>; Tomer Bar @meta.com>; Naureen @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG Thanks, coming out of IXFN review, it would be helpful to understand: 1. Root cause of classifier not detecting the incident 2. Likelihood of error reoccurring with current classifier 3. Cost to get classifier to min acceptable performance (if not today) From: Ashley @meta.com> To: Sridhar Date: Friday, August 11, 2023 at 5:27 PM > @meta.com>, Sayed Otaru < @meta.com>, Arcadiy @meta.com>, Piyush @meta.com>, Rocky Bar < @meta.com>, Emily Dalton Smith < @meta.com>, Amit Bhattacharyya @meta.com>, Kate Gotimer < @meta.com>, Arjun @meta.com>, Amal @meta.com>, Naureen Subject: Re: [Heads Up] Bosnia murder live-streamed on IG Thanks Amit. Adding @Naureen who is going to follow up with the team re IXFN review. From: Sridhar @meta.com> Sent: Friday, August 11, 2023 11:40 AM To: Sayed Otaru < @meta.com>; Amit Bhattacharyya < < @meta.com>; Kate Gotimer < @meta.com>; Ashley @meta.com>; Rocky @meta.com>; Emily Dalton Smith < @meta.com>, Gal @meta.com>, Tomer @meta.com> @meta.com>; Arcadiy @meta.com>; Gal @meta.com>; Piyush @meta.com>; Arjun @meta.com>; Amal @meta.com>; Tomer Bar @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG + @Tomer Bar, @Emily Dalton Smith as FYI on this issue. Rocky is working with XI to see why this classifier did not detect this incident. From: Sayed Otaru < @meta.com> Sent: Friday, August 11, 2023 11:06 AM To: Amit Bhattacharyya < @meta.com>; Arcadiy @meta.com>; Kate Gotimer @meta.com>; Gal @meta.com>; Piyush @meta.com>; Ashley @meta.com>; Arjun @meta.com>; Rocky @meta.com>; Sridhar @meta.com>; Amal @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG + our IG Live team @Rocky @Sridhar @Amal Rocky's just confirmed that the FPS classifier was launched on IG in June. Our teams will work together to understand why we did not detect this. CONFIDENTIAL CONFIDENTIAL PLTF EX-04322_0004 ΜΕΤΑ3047MDL-278-00049221 METANMAG-130-00049221 Amit, we agree that there are some significant Integrity risks on the IG Live - some of which we are already in deliberation with XI on (for GC improvements on Live). Following this incident and the earlier incident in Kentucky, I'll connect again with IG Sharing and IG Leads to review our tradeoffs here. From: Amit Bhattacharyya @meta.com> Sent: Friday, August 11, 2023 10:04 AM To: Arcadiy @meta.com>; Kate Gotimer @meta.com>; Gal @meta.com>; Sayed Otaru @meta.com>; Ashley @meta.com>; Piyush @meta.com>; Arjun @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG Adding @Sayed. Have there been serious discussions to deprecate IG Live given the integrity gaps. @Ashley should we do a review of IG Live and FB Live from IXFN? @meta.com>, Gal @meta.com>, From: Amit Bhattacharyya < Date: Friday, August 11, 2023 at 10:01 AM To: Arcadiy Piyush < @meta.com> @meta.com>, Kate Gotimer @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG So it seems we still don't have an IG Live classifier for First Person Shooter videos From: Amit Bhattacharyya < Date: Friday, August 11, 2023 at 9:45 AM To: Arcadiy Piyush < @meta.com> @meta.com>, Kate Gotimer < @meta.com>, Gal @meta.com>, @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG What I want to understand: 1/ why we think our live video models didn't detect it if there was gunshot, comments etc. of murder. Are these models still working and do we test them? 2/ understand if MAD was the correct response/ should we have deleted the video From: Arcadiy @meta.com> Date: Friday, August 11, 2023 at 9:29 AM To: Amit Bhattacharyya < @meta.com>, Piyush @meta.com>, Kate Gotimer @meta.com>, Gal @meta.com> Subject: Re: [Heads Up] Bosnia murder live-streamed on IG < + @Gal @Piyush From: Amit Bhattacharyya < Date: Friday, August 11, 2023 at 09:25 To: Kate Gotimer < @meta.com> @meta.com>, Arcadiy Subject: Fwd: [Heads Up] Bosnia murder live-streamed on IG Do we know why we missed SLA of 5 mins? CONFIDENTIAL CONFIDENTIAL @meta.com> PLTF EX-04322_0005 ΜΕΤΑ3047MDL-278-00049222 METANMAG-130-00049222 From: Rori @meta.com> Sent: Friday, August 11, 2023 9:17 AM To: Nick Clegg < @meta.com>; Joel Kaplan < @meta.com>; Monika Bickert @meta.com>; Jennifer Newstead < @meta.com>; Chris Sonderby @meta.com> Cc: Nell McCarthy @meta.com>; Neil Potts @meta.com>; Andy O'Connell @meta.com>; Molly Cutler < @meta.com>; David Ginsberg @meta.com>; Chris Norton @meta.com>; Erica @meta.com>; Acadia @meta.com>; Scott @meta.com>; Emily @meta.com>; Cristian @meta.com>; Richard @meta.com>; Guy Rosen @meta.com>; Amit Bhattacharyya @meta.com>; Lee Freedman @meta.com>; Alison @meta.com>; Mike @meta.com>; John DeVine @meta.com>; James Mitchell < @meta.com>; Jessica @meta.com>; David < @meta.com>; Laura @meta.com>; Richard @meta.com>; Strategic Response Policy @meta.com>; Markus Reinisch @meta.com>; Jacob @meta.com>; Clare @meta.com>; Klaus @meta.com>; Adam Mosseri @meta.com>; Tara @meta.com>; Liz @meta.com>; Ben @meta.com> Subject: [Heads Up] Bosnia murder live-streamed on IG EMAIL CONTAINS SENSITIVE LAW ENFORCEMENT INFORMATION: DO NOT FORWARD All, We are making you aware that earlier today a man in Bosnia live-streamed the murder of a woman on IG and claimed to have murdered at least two other people, after which he reportedly took his own life. No action is requested from Leadership. Based on the questions received by Comms from Radio Free Europe it is possible Meta may be criticised for the length of time the live stream remained on-platform (approximately three hours). There is no evidence that automation detected the video prior to removal and operational teams do not believe it should have based on the footage. Situation • • The man live-streamed himself shooting his ex-wife in the head, whilst a child could be heard screaming in the background, before he streamed a second video approximately three hours later in which he claimed to have killed at least two other people (further details below). Police in the town of Tuzla identified the man as Nermin Sulejmanovic and said he had killed three people and injured three others before killing himself on Friday morning. Bosnian Prime Minister Nermin Niksic said he had "no words to describe what happened" and police have not released any details on a potential motivation for the violence. The incident has been widely covered in local and international media. Comms has received two media enquiries (Vice and Radio Free Europe) at the time of writing - though Vice has already published and Meta is not the focus. Critical Information Gaps • We are unaware of any motive for the killing but authorities do not believe it to be terrorism related at this time. None of the victims have been identified so we do not know if they have any on platform presence. XFN Response • Event Designation: Core Policy Team (CPT) has designated the incident as Multiple Victim Violence (MVV) so we will remove all praise, support and representation (PSR) of the event and/or perpetrator(s) from our platforms. Live Video Details: CONFIDENTIAL CONFIDENTIAL PLTF EX-04322_0006 ΜΕΤΑ3047MDL-278-00049223 METANMAG-130-00049223 Ο Ο The video of the user shooting the woman in the head was reported 1,071 times by users, and had 32K VPVs at the point it was removed 2 hours and 58 minutes after being posted. There is no evidence that automation detected the video prior to removal and operational teams do not believe automation should have identified it based on the footage. • Live Video Timeline: • • • • • Ο Ο Ο The user began a live stream on IG at 09:32 (UK) which lasted for five minutes and 11 seconds (this is the video in which he is seen shooting a woman in the head). The first user report was received at 09:51 which was correctly actioned by a reviewer who applied a Mark-as-Disturbing (MAD) interstitial as the imagery did not include any visible innards or other violations per the Graphic Violence policy (the footage did not violate the DOI policy at the time of review because the incident had not been designated MVV at that point). Outsourcing (OS) escalated the video to Global Response Operations (GRO) at 11:00, after identifying a significant number of reports for the same content in their queues. GRO in turn escalated it to PResc at 11:31 requesting an exception to remove the video, and it was triaged to Community Risk Prevention (CORP) for action at 11:42. CORP requested Content Policy input at 12:16 and the content was removed at 12:30 after an escalations only policy was applied that states content should be deleted when there is a depiction of the moment a person is killed and there is a media report confirming the death has taken place. Content Policy designated the incident as MVV shortly afterward which rendered the original video inherently violating under the DOI Policy and also led to the perpetrator's on platform accounts being disabled. The user posted a further live stream video beginning at 12:25 (UK) and lasting 11 minutes and 48 seconds in which he admitted to the murder of two other people. The video was live on IG for 17 minutes prior to being removed when the account was disabled at 12:53. The video was not reported by any users prior to the account being disabled. Operational Enforcement: Ο Ο Perpetrator Accounts: Escalations disabled two IG accounts and two FB accounts belonging to the perpetrator. Ops is investigating the accounts but have not identified any indication of planning or coordination. Footage: Escalations have banked two versions of the live-stream in which the user shoots the woman for removal including on private surfaces (i.e. Messenger and IG Direct), as well as the live-stream in which the user admits to the murder of two other people. GIFCT members are aware and are tracking the incident. SRP confirmed that we have removed video and the suspects account, and engaged with GIFCT to confirm that it does not meet the threshold for launching GIFCT's Content Incident Protocol as there is no ongoing threat or observed spread of the video on member platforms. LE Outreach is in contact with local law enforcement, and they have made one data disclosure request relating to the perpetrator's IG account. Government, Politics & Non-profits (GPN) received an enquiry from a contact at the Bosnian Government Chancellory at 12:07 (UK) requesting the video be removed. Media Enquiries: Ο Ο CONFIDENTIAL CONFIDENTIAL Comms has received two media enquiries (one from Vice Media and one from Radio Free Europe). The incident has also been covered widely by other international media, including AP, without approaching us for comment. The following draft reactive statement has been prepared to respond to enquiries - though Comms is being selective about sharing, given in many articles we are not the focus of criticism: "This was a horrific crime and our deepest sympathies are with the victims and their loved ones. We've removed the video, taken steps to prevent possible copies being re- shared, and removed the perpetrator's accounts across both Facebook and Instagram. PLTF EX-04322_0007 ΜΕΤΑ3047MDL-278-00049224 METANMAG-130-00049224 We're also responding to valid legal requests to help support police in Bosnia with their investigations." - A Meta spokesperson • Master Case: C#3941381 We will send a further update when there is additional information, or we have identified risks to Meta. If you have any questions or have additional information the XFN needs to know of, please direct to @meta.com for attention. Thanks Rori ∞ Meta Rori Public Policy Manager - MENA Lead | Strategic Response Policy Meta | Mobile CONFIDENTIAL CONFIDENTIAL PLTF EX-04322_0008 ΜΕΤΑ3047MDL-278-00049225 METANMAG-130-00049225